-- Logs begin at Fri 2022-10-28 08:59:30 UTC, end at Fri 2022-10-28 10:02:17 UTC. -- Oct 28 09:29:44.711123 dv-cinder-zed-3-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Oct 28 09:29:44.719038 dv-cinder-zed-3-iscsi devstack@keystone.service[81563]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Oct 28 09:29:44.721615 dv-cinder-zed-3-iscsi devstack@keystone.service[81563]: *** Starting uWSGI 2.0.18-debian (64bit) on [Fri Oct 28 09:29:44 2022] *** Oct 28 09:29:44.721754 dv-cinder-zed-3-iscsi devstack@keystone.service[81563]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Oct 28 09:29:44.721858 dv-cinder-zed-3-iscsi devstack@keystone.service[81563]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Oct 28 09:29:44.721967 dv-cinder-zed-3-iscsi devstack@keystone.service[81563]: nodename: dv-cinder-zed-3-iscsi Oct 28 09:29:44.722061 dv-cinder-zed-3-iscsi devstack@keystone.service[81563]: machine: x86_64 Oct 28 09:29:44.722164 dv-cinder-zed-3-iscsi devstack@keystone.service[81563]: clock source: unix Oct 28 09:29:44.722262 dv-cinder-zed-3-iscsi devstack@keystone.service[81563]: pcre jit disabled Oct 28 09:29:44.722354 dv-cinder-zed-3-iscsi devstack@keystone.service[81563]: detected number of CPU cores: 4 Oct 28 09:29:44.722486 dv-cinder-zed-3-iscsi devstack@keystone.service[81563]: current working directory: / Oct 28 09:29:44.722588 dv-cinder-zed-3-iscsi devstack@keystone.service[81563]: detected binary path: /usr/bin/uwsgi-core Oct 28 09:29:44.722698 dv-cinder-zed-3-iscsi devstack@keystone.service[81563]: your processes number limit is 47680 Oct 28 09:29:44.722800 dv-cinder-zed-3-iscsi devstack@keystone.service[81563]: your memory page size is 4096 bytes Oct 28 09:29:44.722900 dv-cinder-zed-3-iscsi devstack@keystone.service[81563]: detected max file descriptor number: 2048 Oct 28 09:29:44.723003 dv-cinder-zed-3-iscsi devstack@keystone.service[81563]: lock engine: pthread robust mutexes Oct 28 09:29:44.723267 dv-cinder-zed-3-iscsi devstack@keystone.service[81563]: thunder lock: enabled Oct 28 09:29:44.723559 dv-cinder-zed-3-iscsi devstack@keystone.service[81563]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Oct 28 09:29:44.723731 dv-cinder-zed-3-iscsi devstack@keystone.service[81563]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Oct 28 09:29:44.763813 dv-cinder-zed-3-iscsi devstack@keystone.service[81563]: Python main interpreter initialized at 0x55aaf356c9c0 Oct 28 09:29:44.763813 dv-cinder-zed-3-iscsi devstack@keystone.service[81563]: python threads support enabled Oct 28 09:29:44.763813 dv-cinder-zed-3-iscsi devstack@keystone.service[81563]: your server socket listen backlog is limited to 100 connections Oct 28 09:29:44.763813 dv-cinder-zed-3-iscsi devstack@keystone.service[81563]: your mercy for graceful operations on workers is 90 seconds Oct 28 09:29:44.764178 dv-cinder-zed-3-iscsi devstack@keystone.service[81563]: mapped 403077 bytes (393 KB) for 2 cores Oct 28 09:29:44.764267 dv-cinder-zed-3-iscsi devstack@keystone.service[81563]: *** Operational MODE: preforking *** Oct 28 09:29:44.764504 dv-cinder-zed-3-iscsi devstack@keystone.service[81563]: *** uWSGI is running in multiple interpreter mode *** Oct 28 09:29:44.764504 dv-cinder-zed-3-iscsi devstack@keystone.service[81563]: spawned uWSGI master process (pid: 81563) Oct 28 09:29:44.764715 dv-cinder-zed-3-iscsi systemd[1]: Started Devstack devstack@keystone.service. Oct 28 09:29:44.766851 dv-cinder-zed-3-iscsi devstack@keystone.service[81563]: spawned uWSGI worker 1 (pid: 81576, cores: 1) Oct 28 09:29:44.766851 dv-cinder-zed-3-iscsi devstack@keystone.service[81563]: spawned uWSGI worker 2 (pid: 81577, cores: 1) Oct 28 09:29:44.766851 dv-cinder-zed-3-iscsi devstack@keystone.service[81563]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Oct 28 09:29:49.026274 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 28 09:29:49.026274 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: from cryptography.utils import int_from_bytes Oct 28 09:29:49.027110 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 28 09:29:49.027110 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: from cryptography.utils import int_from_bytes Oct 28 09:29:49.027557 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 28 09:29:49.027557 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: from cryptography.utils import int_from_bytes Oct 28 09:29:49.028017 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Oct 28 09:29:49.028017 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: from cryptography.utils import int_from_bytes Oct 28 09:29:49.076950 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Oct 28 09:29:49.077148 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Oct 28 09:29:49.077317 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] command line args: [] {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Oct 28 09:29:49.077583 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Oct 28 09:29:49.077761 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ================================================================================ {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Oct 28 09:29:49.078017 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] admin_token = **** {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.078101 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Oct 28 09:29:49.078229 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Oct 28 09:29:49.078315 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] config_dir = [] {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.078467 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.078559 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] command line args: [] {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Oct 28 09:29:49.078638 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] config_source = [] {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.078796 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Oct 28 09:29:49.078882 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] debug = True {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.078982 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ================================================================================ {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Oct 28 09:29:49.079063 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.079261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] admin_token = **** {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.079391 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.079481 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] config_dir = [] {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.079543 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.079672 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.079787 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.079895 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] config_source = [] {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.079995 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.080102 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] debug = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.080203 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] list_limit = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.080302 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.080432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] log_config_append = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.080527 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.080630 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.080724 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.080823 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] log_dir = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.080920 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.081019 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] log_file = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.081118 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.081224 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.081348 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] list_limit = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.081428 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.081544 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] log_config_append = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.081624 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.081702 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.081831 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.081909 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] log_dir = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.082030 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.082111 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] log_file = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.082227 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.082311 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.082449 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.082524 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.082638 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.082723 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.082795 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.082948 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.083031 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.083132 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.083242 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.083329 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.083472 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.083598 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.083722 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.083801 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.083951 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.084032 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.084222 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.084312 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.084455 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.084691 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.084773 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.25/identity {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.084884 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] publish_errors = False {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.084971 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.085118 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.085256 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.085335 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.085446 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.085536 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.085747 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.085936 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.25/identity {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.086064 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.086149 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] publish_errors = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.086221 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] standard_threads = False {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.086347 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.086469 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.086608 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.086687 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.086791 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.086874 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.086996 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] use_journal = False {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.087073 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.087249 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] use_json = False {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.087330 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.087424 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] use_stderr = False {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.087547 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] standard_threads = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.087624 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] use_syslog = False {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.087781 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.087929 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.088032 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.088111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.088212 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.088295 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.088392 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] use_journal = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.088528 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.088607 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] use_json = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.088736 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] use_stderr = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.088812 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.088915 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] use_syslog = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.089037 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.089148 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Oct 28 09:29:49.089228 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.089408 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.089491 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.089655 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.089732 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.089905 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.089989 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.090148 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.090224 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.090409 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.090489 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.090645 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.090731 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.090891 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.090969 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.091124 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.091201 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.091369 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.091456 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.091615 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.091694 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.091850 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.091932 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.092084 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.092172 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.092338 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.092415 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.092579 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.092656 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.092810 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.092887 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.093060 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.093136 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.093298 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.093380 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.093556 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.093640 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.093806 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.093889 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.094035 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.094121 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.094265 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.094341 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.094517 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.094596 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.094750 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.094840 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.094996 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.095074 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.095240 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.095318 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.095468 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.095554 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.095697 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.095781 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.095942 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.096020 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.096180 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.096265 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.096412 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.096489 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.096655 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.096733 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.096883 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.096969 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.097115 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.097198 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.097383 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.097467 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.097571 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.097662 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.097799 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.097877 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.098025 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.098102 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.098251 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.098337 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.098508 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.098589 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.098753 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.098832 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.098982 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.099137 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.099211 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.099280 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.099432 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.099510 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.099661 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.099750 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.099893 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.099972 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.100139 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.100223 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.100367 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.100453 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.100600 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.100678 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.100831 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.100907 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.101062 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.101150 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.101295 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.101373 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.101527 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.101604 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.101755 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.101842 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.101993 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.102075 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.102222 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.102310 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.102478 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.102559 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.103234 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.103359 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.103459 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.103552 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.103696 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.103810 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.103995 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.104110 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.104241 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.104318 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.104445 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.104562 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.104648 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.104765 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.104871 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.105000 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.105144 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.105276 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.105408 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.105540 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.105663 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.105791 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.105913 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.106036 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.106167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.106304 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.106458 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.106592 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.106670 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.106745 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.106826 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.106898 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.106963 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.107032 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.107152 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.107248 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.107396 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.107476 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.107630 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.107709 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.107862 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.107942 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.108095 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.108169 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.108326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.108404 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.108556 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.108640 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.108813 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.108899 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.109050 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.109127 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.109293 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.109371 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.109530 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.109610 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.109765 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.109844 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.110002 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.110078 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.110235 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.110313 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.110501 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.110585 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.110738 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.110815 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.110965 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.111046 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.111184 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.111262 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.111412 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.111503 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.111643 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.111730 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.111872 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.111950 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.112111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.112196 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.112342 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.112429 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.112578 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.112662 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.112810 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.112887 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.113044 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.113133 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.113278 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.113362 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.113522 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.113600 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.113758 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] auth.external = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.113846 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.113986 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.114064 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.114230 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.114336 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.114516 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.114598 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.114746 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] auth.password = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.114836 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.114975 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] auth.token = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.115054 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.115212 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.115290 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] auth.external = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.115443 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.115530 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.115687 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.115766 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.115915 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.116002 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.116159 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.116250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] auth.password = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.116405 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.116479 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] auth.token = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.116632 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.116700 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.116864 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] credential.caching = True {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.116950 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.117106 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.117184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.117340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.117416 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.117583 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.117668 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.117825 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.117910 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.118058 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.118136 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.118310 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.118972 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] credential.caching = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.119126 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.119258 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.119380 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.119460 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.119612 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.119707 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.119865 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.119944 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.120114 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.120198 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.120354 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.120441 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.120605 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.120682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.120855 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.120933 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.121093 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.121178 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.121329 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] federation.caching = True {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.121405 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.121563 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.121647 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.121807 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.121893 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.122050 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.122134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.122278 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.122355 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.122554 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.122647 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.122791 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.122869 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] federation.caching = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.123043 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.123137 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.123282 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.123445 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.123524 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.123597 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.123764 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.123844 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.124012 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.124093 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.124245 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] identity.caching = True {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.124324 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.124489 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.124568 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.124729 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.124810 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.124962 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.125040 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.125195 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.125275 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.125422 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.125499 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.125653 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.125732 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] identity.caching = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.125889 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.125968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.126134 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.126216 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.126380 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.126518 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.126641 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.126741 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.126858 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.126945 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.127085 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.127164 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.127315 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.127394 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.127556 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.127644 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.127797 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.127876 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.128029 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.128107 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.128268 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.128350 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.128503 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.128582 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.128752 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.128838 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.128984 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.129063 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.129211 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.129290 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.129444 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.129523 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.129671 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.129752 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.129902 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.129982 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.130137 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.130216 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.130374 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.130523 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.130651 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.130731 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.130877 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.130955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.131119 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.131200 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.131360 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.131440 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.131592 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.131670 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.131832 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.131912 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.132071 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.132205 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.132285 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.132357 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.132520 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.132599 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.132751 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.132854 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.132982 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.133062 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.133215 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.133295 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.133451 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.133529 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.133690 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.133769 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.133981 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.134115 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.134239 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.134341 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.134444 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.134592 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.134681 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.134823 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.134922 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.135052 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.135167 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.135300 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.135537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.135615 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.135769 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.135846 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.136002 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.136088 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.136235 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.136313 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.136465 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.user = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.136542 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.136703 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.136789 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.136958 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.137035 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.137186 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.137262 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.137426 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.137510 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.137668 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.137746 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.137904 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.user = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.137981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.138135 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.138220 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.138363 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.138473 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.138619 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.138698 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.138852 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.138943 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.139086 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.139167 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.139314 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.139392 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.139557 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.139654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.139803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.139886 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.140039 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.140116 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.140277 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.140364 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.140518 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.140596 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.140744 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.140820 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.140990 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.141076 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.141227 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.141306 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.141462 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.141540 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.141695 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.141781 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.141927 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.142013 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.142162 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.142239 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.142426 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.142519 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.142664 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.142741 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.142904 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.142983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.143132 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.143217 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.143369 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.143446 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.143600 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.143677 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.143835 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.143922 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.144069 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.144147 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.144311 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.144388 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.144552 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.144681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.144763 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.144833 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.145000 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.145079 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.145235 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] resource.caching = True {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.145321 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.145478 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.145554 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.145720 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.145797 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.145952 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.146044 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.146186 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.146263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.146444 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.146527 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.146677 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.146767 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] resource.caching = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.146920 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.146998 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.147155 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.147234 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.147393 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.147480 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.147625 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] role.caching = True {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.147702 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.147866 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] role.driver = sql {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.147943 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.148096 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.148260 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.148336 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.148406 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.148570 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.148648 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.148811 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.148890 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.149047 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] role.caching = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.149124 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.149287 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.149365 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] role.driver = sql {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.149554 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.149632 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.149799 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.149876 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.150044 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.150122 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.150276 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.150356 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.150555 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.150633 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.150800 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.150876 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.151042 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.151135 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.151289 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.151366 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.151541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.151712 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.151798 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.151949 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.152027 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.152195 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.152273 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.152442 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.152528 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.152687 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.152765 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.152933 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.153127 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.153204 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.153369 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.153451 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.153624 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.153702 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.153870 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.153957 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.154102 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.154181 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.154331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.154445 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.154585 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.154677 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.154835 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.154913 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.155087 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.155165 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.155327 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.155412 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.155562 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.155651 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.155790 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.155867 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.156015 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.156101 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.156246 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] token.caching = True {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.156324 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.156734 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.156816 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.156978 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.157065 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.157212 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.157290 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.157457 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.157541 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.157692 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.157779 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.157923 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.158000 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] token.caching = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.158159 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.158235 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.158414 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.158522 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.158679 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.158765 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.158893 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.158971 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.159128 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.159215 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.159358 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.159437 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.159599 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.159677 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.159850 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.159934 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.160086 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.160171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.160327 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.160407 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.160566 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.160642 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.160804 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.160884 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] database.connection = **** {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.161055 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.161133 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.161295 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.161372 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.161531 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.161610 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.161766 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.161859 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.161996 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.162081 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.162229 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.162309 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] database.connection = **** {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.162485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.162569 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.162721 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.162799 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.162962 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.163041 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.163195 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.163275 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.163434 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.163511 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.163672 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.163774 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.163913 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.163994 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.164144 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.164222 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.164384 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.164462 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.164611 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.164691 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.164847 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.164933 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.165077 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.165640 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.165758 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81576) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Oct 28 09:29:49.165860 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.166103 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.166353 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.166612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.166859 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.167100 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Oct 28 09:29:49.167274 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81577) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Oct 28 09:29:49.195124 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81576) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.195351 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81576) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.195578 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81576) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.195831 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.196070 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.196301 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.204534 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81576) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.205288 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.232943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81576) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.233167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81576) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.233720 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.233958 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.286267 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81576) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.286867 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.334856 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81576) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.335088 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81576) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.335875 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.336124 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.491637 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.491944 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.492075 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.492370 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81576) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.492570 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.492701 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81576) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.492829 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.492947 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81576) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.493062 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.493176 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81576) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.493176 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81576) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.493400 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81576) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.516486 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.516723 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.516813 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81576) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.516960 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.517044 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81576) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.517193 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.517277 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81576) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.517423 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.517514 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81576) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.517648 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.517739 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81576) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.517862 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81576) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.517943 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.518079 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81576) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.529796 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.529933 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81576) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.543430 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.543594 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81576) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.543679 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.543741 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81576) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.543910 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.544048 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81576) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.544173 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.544303 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81576) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.544431 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.544559 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81576) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.544693 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.544822 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81576) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.544950 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81577) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.545092 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81576) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Oct 28 09:29:49.561438 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.561658 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 28 09:29:49.561787 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.561911 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 28 09:29:49.562040 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.562168 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 28 09:29:49.562295 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.562462 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Oct 28 09:29:49.562588 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.562717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.562857 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.562983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.563119 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.563247 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.563375 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.563503 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.573105 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.573398 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.573672 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.573858 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.573968 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.574329 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.575273 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.575273 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.581057 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81576) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 09:29:49.583462 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81577) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 09:29:49.583989 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81576) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 09:29:49.584272 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.584555 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.584838 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.585129 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.585391 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.585661 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.585936 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.586206 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.586423 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81577) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 09:29:49.586508 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.586713 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.586808 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.587512 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.587887 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.588179 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.588444 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.588718 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.588999 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.589274 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.589550 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.589822 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.610776 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.612253 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81576) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 09:29:49.612539 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.613163 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.614691 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81577) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 09:29:49.614984 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.617217 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81576) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 09:29:49.617502 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.617785 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.619736 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81577) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 09:29:49.620028 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.620314 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.624237 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81576) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 09:29:49.624530 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.626733 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81577) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 09:29:49.627027 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.630291 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81576) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 09:29:49.630607 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.630894 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.631177 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.631455 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.631732 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.632002 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.632271 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.632876 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81577) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 09:29:49.633184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.633481 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.633771 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.634098 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.634444 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.634915 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.635387 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.647835 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.648127 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.652522 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81576) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 09:29:49.653548 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.653837 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.655845 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.656144 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.658023 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81577) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 09:29:49.660825 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81576) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 09:29:49.661346 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.661656 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.663956 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81576) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 09:29:49.665893 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81577) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 09:29:49.667564 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.667866 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.668152 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.668434 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.668720 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.669006 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.669095 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81577) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 09:29:49.672229 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.672536 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.672829 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.673120 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.673403 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.673688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.685259 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81576) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 09:29:49.685549 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.685818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.686080 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.689961 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81577) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 09:29:49.690256 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.690553 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.690824 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.692907 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.694469 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.696007 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.696282 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.697784 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.699059 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81576) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 09:29:49.699342 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.699432 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.699630 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.699914 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.700300 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.700920 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.701201 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.703904 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81577) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 09:29:49.704195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.704491 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.704780 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.705169 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.711460 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81576) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 09:29:49.711748 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.712558 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.712827 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.713206 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.713501 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.713781 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.716938 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81577) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 09:29:49.717231 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.717516 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.717798 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.718180 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.718496 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.718784 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.729175 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81576) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 09:29:49.732091 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81576) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 09:29:49.734320 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81577) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 09:29:49.735003 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.736444 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.737288 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81577) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 09:29:49.737926 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81576) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 09:29:49.738212 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.738631 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.740230 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.741700 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.743203 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81577) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 09:29:49.744007 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.744429 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.745502 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.746988 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81576) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 09:29:49.749875 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.750269 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.750578 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.750843 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.750938 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.752339 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81577) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 09:29:49.755263 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.755662 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.755952 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.756236 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.758653 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81576) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 09:29:49.758972 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.759296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.764167 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81577) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 09:29:49.764476 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.764773 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.766175 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81576) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 09:29:49.766485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.766770 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.767046 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.767316 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.767594 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.767865 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.768149 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.768419 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.768692 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.768975 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.769263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.769562 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.769842 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81576) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.771732 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81577) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Oct 28 09:29:49.772021 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.772304 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.772585 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.772861 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.773138 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.773415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.773702 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.773978 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.774250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.775102 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.775392 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.775664 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.775954 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81577) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Oct 28 09:29:49.804530 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x55aaf356c9c0 pid: 81576 (default app) Oct 28 09:29:49.807834 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8ec400-f0f4-44c2-9421-de68fcec84b8 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:29:49.808121 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8ec400-f0f4-44c2-9421-de68fcec84b8 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:29:49.808392 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8ec400-f0f4-44c2-9421-de68fcec84b8 None None] PATH_INFO: `/v3` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:29:49.810455 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x55aaf356c9c0 pid: 81577 (default app) Oct 28 09:29:49.812537 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1/1] 10.222.0.25 () {50 vars in 791 bytes} [Fri Oct 28 09:29:49 2022] GET /identity/v3/ => generated 255 bytes in 7 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:29:56.622181 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd1a654-8632-4f39-83aa-4504a13beba8 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:29:56.622713 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd1a654-8632-4f39-83aa-4504a13beba8 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:29:56.622820 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd1a654-8632-4f39-83aa-4504a13beba8 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:29:56.627004 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1/2] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:29:56 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 09:29:56.632869 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d89ba7-52be-4ed8-a798-58247e1c92e6 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:29:56.633311 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d89ba7-52be-4ed8-a798-58247e1c92e6 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:29:56.633446 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d89ba7-52be-4ed8-a798-58247e1c92e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:29:56.705190 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: INFO dbcounter [None req-93d89ba7-52be-4ed8-a798-58247e1c92e6 None None] Registered counter for database keystone Oct 28 09:29:56.726794 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG oslo_db.sqlalchemy.engines [None req-93d89ba7-52be-4ed8-a798-58247e1c92e6 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81576) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Oct 28 09:29:56.732916 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writer thread running {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Oct 28 09:29:56.915755 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG passlib.handlers.bcrypt [None req-93d89ba7-52be-4ed8-a798-58247e1c92e6 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81576) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Oct 28 09:29:56.916140 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG passlib.handlers.bcrypt [None req-93d89ba7-52be-4ed8-a798-58247e1c92e6 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81576) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Oct 28 09:29:56.974500 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-93d89ba7-52be-4ed8-a798-58247e1c92e6 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:29:56.976135 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-93d89ba7-52be-4ed8-a798-58247e1c92e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:29:57.108200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 2/3] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:29:56 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 476 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Oct 28 09:29:57.577031 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd1a654-8632-4f39-83aa-4504a13beba8 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:29:57.646917 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: INFO dbcounter [None req-5dd1a654-8632-4f39-83aa-4504a13beba8 None None] Registered counter for database keystone Oct 28 09:29:57.669658 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG oslo_db.sqlalchemy.engines [None req-5dd1a654-8632-4f39-83aa-4504a13beba8 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81577) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Oct 28 09:29:57.676064 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writer thread running {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Oct 28 09:29:57.798866 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd1a654-8632-4f39-83aa-4504a13beba8 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:29:57.803132 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-586493e4-33ab-4193-a1ea-e8314fe49c75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:29:57.803769 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586493e4-33ab-4193-a1ea-e8314fe49c75 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:29:57.804053 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586493e4-33ab-4193-a1ea-e8314fe49c75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:29:57.804329 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-586493e4-33ab-4193-a1ea-e8314fe49c75 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:29:57.812995 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-586493e4-33ab-4193-a1ea-e8314fe49c75 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:29:57.864446 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: WARNING keystone.common.rbac_enforcer.enforcer [None req-586493e4-33ab-4193-a1ea-e8314fe49c75 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Oct 28 09:29:57.864798 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-586493e4-33ab-4193-a1ea-e8314fe49c75 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:29:57.875601 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 2/4] 10.222.0.25 () {58 vars in 1324 bytes} [Fri Oct 28 09:29:57 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 299 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 28 09:29:57.881563 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d89ba7-52be-4ed8-a798-58247e1c92e6 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:29:57.898288 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d89ba7-52be-4ed8-a798-58247e1c92e6 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:29:57.903242 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-306605a7-3f63-43a9-a60c-a9c406378362 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:29:57.904364 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306605a7-3f63-43a9-a60c-a9c406378362 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:29:57.904647 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306605a7-3f63-43a9-a60c-a9c406378362 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:29:57.904921 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306605a7-3f63-43a9-a60c-a9c406378362 admin admin] PATH_INFO: `/v3/projects/ac54d153685c4b2288a7ed215d0e24d2` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:29:57.906810 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-306605a7-3f63-43a9-a60c-a9c406378362 admin admin] RBAC: Authorizing `identity:get_project(project_id=ac54d153685c4b2288a7ed215d0e24d2)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:29:57.959215 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: WARNING keystone.common.rbac_enforcer.enforcer [None req-306605a7-3f63-43a9-a60c-a9c406378362 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Oct 28 09:29:57.959569 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-306605a7-3f63-43a9-a60c-a9c406378362 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:29:57.964723 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 3/5] 10.222.0.25 () {56 vars in 1207 bytes} [Fri Oct 28 09:29:57 2022] GET /identity/v3/projects/ac54d153685c4b2288a7ed215d0e24d2 => generated 344 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:00.089745 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afeb5271-0596-436c-9726-de823f5da5db None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:00.090351 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afeb5271-0596-436c-9726-de823f5da5db None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:00.090351 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afeb5271-0596-436c-9726-de823f5da5db None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:00.094359 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 3/6] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:00 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:30:00.099736 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d635937e-a6be-4eee-9aa0-f054d944fbcb None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:00.100125 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d635937e-a6be-4eee-9aa0-f054d944fbcb None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:00.100289 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d635937e-a6be-4eee-9aa0-f054d944fbcb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:00.121033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-d635937e-a6be-4eee-9aa0-f054d944fbcb None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:00.122586 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-d635937e-a6be-4eee-9aa0-f054d944fbcb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:00.130275 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 4/7] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:00 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Oct 28 09:30:00.396256 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afeb5271-0596-436c-9726-de823f5da5db None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:00.408536 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afeb5271-0596-436c-9726-de823f5da5db None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:00.412702 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94507f4e-7d4e-42df-9dd1-d0ea70b28c12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:00.413297 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94507f4e-7d4e-42df-9dd1-d0ea70b28c12 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:00.413579 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94507f4e-7d4e-42df-9dd1-d0ea70b28c12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:00.413855 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94507f4e-7d4e-42df-9dd1-d0ea70b28c12 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:00.421954 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94507f4e-7d4e-42df-9dd1-d0ea70b28c12 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:00.423735 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94507f4e-7d4e-42df-9dd1-d0ea70b28c12 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:00.432448 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 4/8] 10.222.0.25 () {58 vars in 1324 bytes} [Fri Oct 28 09:30:00 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 28 09:30:00.437950 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d635937e-a6be-4eee-9aa0-f054d944fbcb None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:00.447521 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d635937e-a6be-4eee-9aa0-f054d944fbcb None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:00.451824 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a56088e-8a35-4f74-8224-2da7f5184e65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:00.452446 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a56088e-8a35-4f74-8224-2da7f5184e65 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:00.452726 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a56088e-8a35-4f74-8224-2da7f5184e65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:00.453000 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a56088e-8a35-4f74-8224-2da7f5184e65 admin admin] PATH_INFO: `/v3/users/8afbf6a033494d4cbbdea7a924ab5c63` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:00.454341 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a56088e-8a35-4f74-8224-2da7f5184e65 admin admin] RBAC: Authorizing `identity:get_user(user_id=8afbf6a033494d4cbbdea7a924ab5c63)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:00.456199 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a56088e-8a35-4f74-8224-2da7f5184e65 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:00.457751 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 5/9] 10.222.0.25 () {56 vars in 1198 bytes} [Fri Oct 28 09:30:00 2022] GET /identity/v3/users/8afbf6a033494d4cbbdea7a924ab5c63 => generated 247 bytes in 20 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:02.678675 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a62a7e-0fde-41ec-aab2-d5cad2c2d6b4 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:02.679217 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a62a7e-0fde-41ec-aab2-d5cad2c2d6b4 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:02.679217 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a62a7e-0fde-41ec-aab2-d5cad2c2d6b4 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:02.680615 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 5/10] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:30:02.685742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e353be-f9b4-47e7-8d85-06bbb025c77d None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:02.686185 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e353be-f9b4-47e7-8d85-06bbb025c77d None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:02.686458 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e353be-f9b4-47e7-8d85-06bbb025c77d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:02.696041 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20cb966b-9eb7-4241-8fe7-2ae23cf77387 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:02.696845 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20cb966b-9eb7-4241-8fe7-2ae23cf77387 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:02.697123 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20cb966b-9eb7-4241-8fe7-2ae23cf77387 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:02.698637 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 6/11] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 09:30:02.703250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24bdd2d-74d4-414b-8887-ab1c34a42d6b None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:02.703537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24bdd2d-74d4-414b-8887-ab1c34a42d6b None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:02.703814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24bdd2d-74d4-414b-8887-ab1c34a42d6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:02.705269 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-e8e353be-f9b4-47e7-8d85-06bbb025c77d None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:02.706504 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-e8e353be-f9b4-47e7-8d85-06bbb025c77d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:02.746002 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG passlib.handlers.bcrypt [None req-a24bdd2d-74d4-414b-8887-ab1c34a42d6b None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81577) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Oct 28 09:30:02.746424 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG passlib.handlers.bcrypt [None req-a24bdd2d-74d4-414b-8887-ab1c34a42d6b None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81577) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Oct 28 09:30:02.790999 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 6/12] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:30:02.796518 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-a24bdd2d-74d4-414b-8887-ab1c34a42d6b None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:02.798160 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a24bdd2d-74d4-414b-8887-ab1c34a42d6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:02.898935 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 7/13] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 09:30:03.065814 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e353be-f9b4-47e7-8d85-06bbb025c77d None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:03.095536 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:03.170971 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a24bdd2d-74d4-414b-8887-ab1c34a42d6b None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:03.189471 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e353be-f9b4-47e7-8d85-06bbb025c77d None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:03.227226 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa9130a-3eb4-4bad-9b6e-a86394076707 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:03.228026 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa9130a-3eb4-4bad-9b6e-a86394076707 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:03.228358 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa9130a-3eb4-4bad-9b6e-a86394076707 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:03.228675 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa9130a-3eb4-4bad-9b6e-a86394076707 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:03.241191 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aa9130a-3eb4-4bad-9b6e-a86394076707 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:30:03.253388 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-7aa9130a-3eb4-4bad-9b6e-a86394076707 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:30:03.253388 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:30:03.253388 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:30:03.253388 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:30:03.253388 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:30:03.253388 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:30:03.253388 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:30:03.253388 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:30:03.253388 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:30:03.253388 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:30:03.253388 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:30:03.253388 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:30:03.253388 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:30:03.253388 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:30:03.253388 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:30:03.253388 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:30:03.254324 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:30:03.254324 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:30:03.254324 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:30:03.254324 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:30:03.254324 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:30:03.254324 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:30:03.254324 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:30:03.254324 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:30:03.254324 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:30:03.254324 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:30:03.254324 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:30:03.254324 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:30:03.254324 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:30:03.254324 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:30:03.254324 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:30:03.254324 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:30:03.255207 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:30:03.255207 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:30:03.255207 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:30:03.255207 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:30:03.255207 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:30:03.255207 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:30:03.255207 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:30:03.255207 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:30:03.255207 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:30:03.255207 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:30:03.255207 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:30:03.255207 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:30:03.255802 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 7/14] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Oct 28 09:30:03 2022] GET /identity/v3/roles/admin => generated 83 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:30:03.263441 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa9130a-3eb4-4bad-9b6e-a86394076707 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:03.286018 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a24bdd2d-74d4-414b-8887-ab1c34a42d6b None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:03.289367 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:03.333887 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff4a36f-5434-4930-9b54-861861b1f3ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:03.334795 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff4a36f-5434-4930-9b54-861861b1f3ab None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:03.335082 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff4a36f-5434-4930-9b54-861861b1f3ab None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:03.335355 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff4a36f-5434-4930-9b54-861861b1f3ab None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:03.353246 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:03.374154 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa9130a-3eb4-4bad-9b6e-a86394076707 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:03.407854 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff4a36f-5434-4930-9b54-861861b1f3ab None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:03.408877 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9656551a-5329-42de-aab7-ffdd588fea36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:03.409363 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff4a36f-5434-4930-9b54-861861b1f3ab None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:03.409632 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9656551a-5329-42de-aab7-ffdd588fea36 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:03.409985 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9656551a-5329-42de-aab7-ffdd588fea36 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:03.410333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9656551a-5329-42de-aab7-ffdd588fea36 None admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:03.411107 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9656551a-5329-42de-aab7-ffdd588fea36 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:03.411485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9656551a-5329-42de-aab7-ffdd588fea36 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:03.412972 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9656551a-5329-42de-aab7-ffdd588fea36 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:03.425377 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 8/15] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:30:03 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:30:03.431565 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9656551a-5329-42de-aab7-ffdd588fea36 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:03.507961 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 8/16] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:30:03 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 28 09:30:03.513574 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff4a36f-5434-4930-9b54-861861b1f3ab None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:03.516407 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:03.530219 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9656551a-5329-42de-aab7-ffdd588fea36 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:03.567698 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5630260-9a8d-4a3a-8957-afbfd8597793 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:03.568177 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5630260-9a8d-4a3a-8957-afbfd8597793 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:03.568459 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5630260-9a8d-4a3a-8957-afbfd8597793 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:03.568735 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5630260-9a8d-4a3a-8957-afbfd8597793 None admin] PATH_INFO: `/v3/users/8afbf6a033494d4cbbdea7a924ab5c63` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:03.569899 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5630260-9a8d-4a3a-8957-afbfd8597793 None admin] RBAC: Authorizing `identity:get_user(user_id=8afbf6a033494d4cbbdea7a924ab5c63)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:03.571501 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5630260-9a8d-4a3a-8957-afbfd8597793 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:03.573077 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 9/17] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:30:03 2022] GET /identity/v3/users/8afbf6a033494d4cbbdea7a924ab5c63 => generated 247 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:30:03.578444 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5630260-9a8d-4a3a-8957-afbfd8597793 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:03.594083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:03.613414 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff4a36f-5434-4930-9b54-861861b1f3ab None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:03.646891 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2933e888-257d-451c-abc9-0a2b22b9e2e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:03.647751 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2933e888-257d-451c-abc9-0a2b22b9e2e0 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:03.648035 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2933e888-257d-451c-abc9-0a2b22b9e2e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:03.648315 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2933e888-257d-451c-abc9-0a2b22b9e2e0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:03.667463 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2933e888-257d-451c-abc9-0a2b22b9e2e0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:03.668950 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2933e888-257d-451c-abc9-0a2b22b9e2e0 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:03.679486 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5630260-9a8d-4a3a-8957-afbfd8597793 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:03.688363 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-2933e888-257d-451c-abc9-0a2b22b9e2e0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:30:03.688363 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:30:03.688363 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:30:03.688363 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:30:03.688363 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:30:03.688363 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:30:03.688363 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:30:03.688363 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:30:03.688363 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:30:03.688363 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:30:03.688363 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:30:03.688363 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:30:03.688363 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:30:03.688363 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:30:03.688363 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:30:03.688363 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:30:03.689750 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:30:03.689750 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:30:03.689750 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:30:03.689750 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:30:03.689750 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:30:03.689750 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:30:03.689750 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:30:03.689750 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:30:03.689750 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:30:03.689750 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:30:03.689750 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:30:03.689750 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:30:03.689750 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:30:03.689750 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:30:03.689750 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:30:03.689750 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:30:03.689750 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:30:03.690850 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:30:03.690850 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:30:03.690850 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:30:03.690850 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 9/18] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:30:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:30:03.695177 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2933e888-257d-451c-abc9-0a2b22b9e2e0 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:03.716476 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b244e49a-5c0b-4cb2-863c-7c20c2cb88c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:03.717108 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b244e49a-5c0b-4cb2-863c-7c20c2cb88c8 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:03.717393 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b244e49a-5c0b-4cb2-863c-7c20c2cb88c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:03.717668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b244e49a-5c0b-4cb2-863c-7c20c2cb88c8 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:03.718852 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b244e49a-5c0b-4cb2-863c-7c20c2cb88c8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:03.720319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b244e49a-5c0b-4cb2-863c-7c20c2cb88c8 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:03.721882 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 10/19] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:30:03 2022] GET /identity/v3/domains/default => generated 199 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:30:03.728131 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b244e49a-5c0b-4cb2-863c-7c20c2cb88c8 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:03.753887 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:03.799531 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2933e888-257d-451c-abc9-0a2b22b9e2e0 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:03.809011 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:03.828621 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b244e49a-5c0b-4cb2-863c-7c20c2cb88c8 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:03.833191 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0de6d1ab-ff5e-4f63-9b7f-db4d60b14633 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:03.833796 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de6d1ab-ff5e-4f63-9b7f-db4d60b14633 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:03.834078 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de6d1ab-ff5e-4f63-9b7f-db4d60b14633 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:03.834352 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0de6d1ab-ff5e-4f63-9b7f-db4d60b14633 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:03.835023 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de6d1ab-ff5e-4f63-9b7f-db4d60b14633 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:03.835333 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de6d1ab-ff5e-4f63-9b7f-db4d60b14633 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:03.836694 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0de6d1ab-ff5e-4f63-9b7f-db4d60b14633 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:03.853991 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 10/20] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:30:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:30:03.861801 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ac0c69-e42a-4cb3-873b-e50512b8e9e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:03.862462 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ac0c69-e42a-4cb3-873b-e50512b8e9e3 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:03.862747 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ac0c69-e42a-4cb3-873b-e50512b8e9e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:03.863025 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ac0c69-e42a-4cb3-873b-e50512b8e9e3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:03.863777 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ac0c69-e42a-4cb3-873b-e50512b8e9e3 None admin] RBAC: Adding query filter params (role.id=e13d798a98ea4c3dbb87a236ec683d32, scope.domain.id=default, user.id=8afbf6a033494d4cbbdea7a924ab5c63) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:03.864091 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ac0c69-e42a-4cb3-873b-e50512b8e9e3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:03.865580 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ac0c69-e42a-4cb3-873b-e50512b8e9e3 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:03.873816 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 11/21] 10.222.0.25 () {56 vars in 1322 bytes} [Fri Oct 28 09:30:03 2022] GET /identity/v3/role_assignments?user.id=8afbf6a033494d4cbbdea7a924ab5c63&scope.domain.id=default&role.id=e13d798a98ea4c3dbb87a236ec683d32 => generated 233 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:30:06.435978 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d5539f-1674-44a9-9533-4983757020ae None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:06.436524 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d5539f-1674-44a9-9533-4983757020ae None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:06.436524 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d5539f-1674-44a9-9533-4983757020ae None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:06.437890 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 11/22] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:30:06.448337 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1289d2f3-8aa0-414a-a975-fdb704bc32b1 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:06.448812 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1289d2f3-8aa0-414a-a975-fdb704bc32b1 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:06.449015 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1289d2f3-8aa0-414a-a975-fdb704bc32b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:06.470830 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-1289d2f3-8aa0-414a-a975-fdb704bc32b1 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:06.472041 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-1289d2f3-8aa0-414a-a975-fdb704bc32b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:06.589400 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 12/23] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:30:06.637437 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f3affb-17a5-4410-9a72-dbc7b0284b14 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:06.637715 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f3affb-17a5-4410-9a72-dbc7b0284b14 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:06.638066 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f3affb-17a5-4410-9a72-dbc7b0284b14 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:06.639679 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 12/24] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:30:06.653657 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca156a7-ced6-497c-9ef6-03dc45330183 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:06.653948 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca156a7-ced6-497c-9ef6-03dc45330183 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:06.654236 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca156a7-ced6-497c-9ef6-03dc45330183 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:06.683784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-bca156a7-ced6-497c-9ef6-03dc45330183 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:06.684996 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-bca156a7-ced6-497c-9ef6-03dc45330183 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:06.772817 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 13/25] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:30:06.945345 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b15e525-f025-498b-903b-5c12cb554709 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:06.945891 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b15e525-f025-498b-903b-5c12cb554709 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:06.946384 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b15e525-f025-498b-903b-5c12cb554709 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:06.947878 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 13/26] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:30:06.953670 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4773d590-fbff-49d8-ad8f-cf7022ec7014 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:06.954130 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4773d590-fbff-49d8-ad8f-cf7022ec7014 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:06.954600 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4773d590-fbff-49d8-ad8f-cf7022ec7014 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:06.963873 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:06.975809 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-4773d590-fbff-49d8-ad8f-cf7022ec7014 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:06.977262 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-4773d590-fbff-49d8-ad8f-cf7022ec7014 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:07.015397 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b15e525-f025-498b-903b-5c12cb554709 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:07.118141 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 14/27] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:30:07.175291 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4773d590-fbff-49d8-ad8f-cf7022ec7014 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:07.223915 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b15e525-f025-498b-903b-5c12cb554709 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:07.257210 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:07.291014 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33bad421-b070-4c3e-ba95-0eb3cd3f370c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:07.291660 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33bad421-b070-4c3e-ba95-0eb3cd3f370c None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:07.291946 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33bad421-b070-4c3e-ba95-0eb3cd3f370c None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:07.292225 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33bad421-b070-4c3e-ba95-0eb3cd3f370c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:07.315879 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33bad421-b070-4c3e-ba95-0eb3cd3f370c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:30:07.326556 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-33bad421-b070-4c3e-ba95-0eb3cd3f370c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:30:07.326556 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:30:07.326556 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:30:07.326556 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:30:07.326556 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:30:07.326556 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:30:07.326556 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:30:07.326556 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:30:07.326556 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:30:07.326556 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:30:07.326556 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:30:07.326556 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:30:07.326556 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:30:07.326556 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:30:07.326556 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:30:07.326556 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:30:07.327444 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:30:07.327444 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:30:07.327444 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:30:07.327444 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:30:07.327444 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:30:07.327444 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:30:07.327444 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:30:07.327444 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:30:07.327444 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:30:07.327444 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:30:07.327444 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:30:07.327444 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:30:07.327444 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:30:07.327444 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:30:07.327444 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:30:07.327444 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:30:07.328202 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:30:07.328202 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:30:07.328202 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:30:07.328202 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:30:07.328202 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:30:07.328202 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:30:07.328202 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:30:07.328202 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:30:07.328202 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:30:07.328202 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:30:07.328202 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:30:07.328202 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:30:07.328202 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 14/28] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Oct 28 09:30:07 2022] GET /identity/v3/roles/admin => generated 83 bytes in 313 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:30:07.329704 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766b7b58-de1c-418c-b497-c260f2160515 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:07.329704 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766b7b58-de1c-418c-b497-c260f2160515 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:07.329704 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766b7b58-de1c-418c-b497-c260f2160515 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:07.336159 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 15/29] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:07 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:30:07.336992 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766b7b58-de1c-418c-b497-c260f2160515 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:07.352810 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4773d590-fbff-49d8-ad8f-cf7022ec7014 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:07.436520 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c74874f7-8751-4c08-b300-367551f5edd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:07.437491 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74874f7-8751-4c08-b300-367551f5edd8 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:07.437992 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74874f7-8751-4c08-b300-367551f5edd8 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:07.439205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74874f7-8751-4c08-b300-367551f5edd8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:07.446724 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:07.465633 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c74874f7-8751-4c08-b300-367551f5edd8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:07.468877 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c74874f7-8751-4c08-b300-367551f5edd8 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:07.504072 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-c74874f7-8751-4c08-b300-367551f5edd8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:30:07.504072 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:30:07.504072 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:30:07.504072 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:30:07.504072 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:30:07.504072 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:30:07.504072 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:30:07.504072 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:30:07.504072 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:30:07.504072 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:30:07.504072 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:30:07.504072 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:30:07.504072 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:30:07.504072 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:30:07.504072 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:30:07.504072 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:30:07.505008 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:30:07.505008 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:30:07.505008 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:30:07.505008 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:30:07.505008 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:30:07.505008 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:30:07.505008 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:30:07.505008 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:30:07.505008 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:30:07.505008 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:30:07.505008 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:30:07.505008 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:30:07.505008 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:30:07.505008 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:30:07.505008 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:30:07.505008 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:30:07.505008 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:30:07.505813 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:30:07.505813 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:30:07.505813 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:30:07.505813 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 15/30] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:30:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 330 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:30:07.506760 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b9e7b0-9dca-46dc-8291-19898ea274f9 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:07.506760 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b9e7b0-9dca-46dc-8291-19898ea274f9 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:07.507408 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b9e7b0-9dca-46dc-8291-19898ea274f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:07.532397 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-f9b9e7b0-9dca-46dc-8291-19898ea274f9 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:07.533068 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766b7b58-de1c-418c-b497-c260f2160515 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:07.534849 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-f9b9e7b0-9dca-46dc-8291-19898ea274f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:07.578996 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0435e7cb-4d37-41da-84f1-cff7bad8589e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:07.579708 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0435e7cb-4d37-41da-84f1-cff7bad8589e None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:07.580165 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0435e7cb-4d37-41da-84f1-cff7bad8589e None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:07.580509 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0435e7cb-4d37-41da-84f1-cff7bad8589e None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:07.581420 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0435e7cb-4d37-41da-84f1-cff7bad8589e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:07.581704 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0435e7cb-4d37-41da-84f1-cff7bad8589e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:07.583230 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0435e7cb-4d37-41da-84f1-cff7bad8589e None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:07.603446 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 16/31] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:30:07 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:07.604521 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0435e7cb-4d37-41da-84f1-cff7bad8589e None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:07.631067 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 16/32] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:30:07.632425 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9b9e7b0-9dca-46dc-8291-19898ea274f9 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:07.665063 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:07.765573 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9b9e7b0-9dca-46dc-8291-19898ea274f9 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:07.769264 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0435e7cb-4d37-41da-84f1-cff7bad8589e None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:07.786824 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:07.809287 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f70114-d72c-4de0-a7a0-b8eb7efab336 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:07.810579 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f70114-d72c-4de0-a7a0-b8eb7efab336 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:07.811101 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f70114-d72c-4de0-a7a0-b8eb7efab336 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:07.811911 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f70114-d72c-4de0-a7a0-b8eb7efab336 None admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:07.813311 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f70114-d72c-4de0-a7a0-b8eb7efab336 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:07.814053 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f70114-d72c-4de0-a7a0-b8eb7efab336 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:07.816526 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f70114-d72c-4de0-a7a0-b8eb7efab336 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:07.837787 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 17/33] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:30:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:07.839127 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f70114-d72c-4de0-a7a0-b8eb7efab336 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:07.885659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e391603-a4cc-4e67-ba81-5d44b4ca5560 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:07.886294 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e391603-a4cc-4e67-ba81-5d44b4ca5560 None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:07.886612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e391603-a4cc-4e67-ba81-5d44b4ca5560 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:07.886892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e391603-a4cc-4e67-ba81-5d44b4ca5560 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:07.887585 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e391603-a4cc-4e67-ba81-5d44b4ca5560 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:07.888988 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e391603-a4cc-4e67-ba81-5d44b4ca5560 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:07.957816 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f70114-d72c-4de0-a7a0-b8eb7efab336 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:07.995498 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08780ca7-fb46-4d84-8f4c-ae656774dd8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:07.996610 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08780ca7-fb46-4d84-8f4c-ae656774dd8f None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:07.997094 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08780ca7-fb46-4d84-8f4c-ae656774dd8f None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:07.997566 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08780ca7-fb46-4d84-8f4c-ae656774dd8f None admin] PATH_INFO: `/v3/users/8afbf6a033494d4cbbdea7a924ab5c63` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:07.999086 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08780ca7-fb46-4d84-8f4c-ae656774dd8f None admin] RBAC: Authorizing `identity:get_user(user_id=8afbf6a033494d4cbbdea7a924ab5c63)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:08.000821 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08780ca7-fb46-4d84-8f4c-ae656774dd8f None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:08.003002 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 18/34] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:30:07 2022] GET /identity/v3/users/8afbf6a033494d4cbbdea7a924ab5c63 => generated 247 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:08.004099 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08780ca7-fb46-4d84-8f4c-ae656774dd8f None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:08.035555 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 17/35] 10.222.0.25 () {60 vars in 1131 bytes} [Fri Oct 28 09:30:07 2022] POST /identity/v3/roles => generated 225 bytes in 431 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:30:08.036933 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0ea60e-6db3-4af1-9153-61cd26b1db04 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:08.037274 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0ea60e-6db3-4af1-9153-61cd26b1db04 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:08.037516 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0ea60e-6db3-4af1-9153-61cd26b1db04 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:08.039033 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 18/36] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:30:08.043951 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0ea60e-6db3-4af1-9153-61cd26b1db04 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:08.100440 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:08.143185 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08780ca7-fb46-4d84-8f4c-ae656774dd8f None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:08.161329 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:08.168226 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0ea60e-6db3-4af1-9153-61cd26b1db04 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:08.185141 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be6b855-e0b9-4ae7-ac57-8c164ce1a979 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:08.185930 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be6b855-e0b9-4ae7-ac57-8c164ce1a979 None admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:08.186288 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be6b855-e0b9-4ae7-ac57-8c164ce1a979 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:08.186627 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be6b855-e0b9-4ae7-ac57-8c164ce1a979 None admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:08.187316 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be6b855-e0b9-4ae7-ac57-8c164ce1a979 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:08.188826 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be6b855-e0b9-4ae7-ac57-8c164ce1a979 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:08.208087 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f49ccce-feb9-4f84-a557-1e29502637dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:08.208698 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f49ccce-feb9-4f84-a557-1e29502637dc None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:08.208992 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f49ccce-feb9-4f84-a557-1e29502637dc None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:08.209514 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f49ccce-feb9-4f84-a557-1e29502637dc None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:08.209956 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f49ccce-feb9-4f84-a557-1e29502637dc None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:08.212197 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f49ccce-feb9-4f84-a557-1e29502637dc None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:08.233570 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 19/37] 10.222.0.25 () {60 vars in 1131 bytes} [Fri Oct 28 09:30:08 2022] POST /identity/v3/roles => generated 219 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:30:08.234793 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f49ccce-feb9-4f84-a557-1e29502637dc None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:08.379482 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f49ccce-feb9-4f84-a557-1e29502637dc None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:08.428055 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f21a4473-2f20-4848-bbdc-2846fd817806 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:08.429072 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21a4473-2f20-4848-bbdc-2846fd817806 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:08.429586 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21a4473-2f20-4848-bbdc-2846fd817806 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:08.430063 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21a4473-2f20-4848-bbdc-2846fd817806 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:08.432361 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f21a4473-2f20-4848-bbdc-2846fd817806 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:08.434121 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f21a4473-2f20-4848-bbdc-2846fd817806 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:08.439809 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 20/38] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:30:08 2022] GET /identity/v3/domains/default => generated 199 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:08.439809 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd37214b-ade8-4d13-a3c4-bebc96e71c99 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:08.439809 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd37214b-ade8-4d13-a3c4-bebc96e71c99 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:08.439809 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd37214b-ade8-4d13-a3c4-bebc96e71c99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:08.462530 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-bd37214b-ade8-4d13-a3c4-bebc96e71c99 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:08.464761 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 19/39] 10.222.0.25 () {60 vars in 1141 bytes} [Fri Oct 28 09:30:08 2022] POST /identity/v3/projects => generated 301 bytes in 459 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:30:08.464915 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-bd37214b-ade8-4d13-a3c4-bebc96e71c99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:08.465002 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a44b3cb-113b-4aa8-9b94-89c62c19d045 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:08.465002 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a44b3cb-113b-4aa8-9b94-89c62c19d045 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:08.465576 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a44b3cb-113b-4aa8-9b94-89c62c19d045 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:08.467720 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 20/40] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:30:08.467720 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a44b3cb-113b-4aa8-9b94-89c62c19d045 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:08.497545 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:08.579083 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 21/41] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:30:08.595380 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d97c70-c7cf-4904-a156-c06f84198f82 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:08.595380 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d97c70-c7cf-4904-a156-c06f84198f82 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:08.595380 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d97c70-c7cf-4904-a156-c06f84198f82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:08.625266 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:08.639159 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a44b3cb-113b-4aa8-9b94-89c62c19d045 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:08.648381 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-90d97c70-c7cf-4904-a156-c06f84198f82 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:08.650281 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-90d97c70-c7cf-4904-a156-c06f84198f82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:08.678804 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47e0e54-b774-4bd7-af84-a282b6955c86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:08.679476 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47e0e54-b774-4bd7-af84-a282b6955c86 None admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:08.679759 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47e0e54-b774-4bd7-af84-a282b6955c86 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:08.680039 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47e0e54-b774-4bd7-af84-a282b6955c86 None admin] PATH_INFO: `/v3/domains/default/users/8afbf6a033494d4cbbdea7a924ab5c63/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:08.683560 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47e0e54-b774-4bd7-af84-a282b6955c86 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=8afbf6a033494d4cbbdea7a924ab5c63, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:08.685375 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47e0e54-b774-4bd7-af84-a282b6955c86 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:08.710445 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 21/42] 10.222.0.25 () {58 vars in 1361 bytes} [Fri Oct 28 09:30:08 2022] PUT /identity/v3/domains/default/users/8afbf6a033494d4cbbdea7a924ab5c63/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 243 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:30:08.763914 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 22/43] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:30:08.876817 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47e0e54-b774-4bd7-af84-a282b6955c86 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:09.009251 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47e0e54-b774-4bd7-af84-a282b6955c86 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:09.040761 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d97c70-c7cf-4904-a156-c06f84198f82 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:09.045555 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974d93f1-af4a-4829-905d-6921e233746b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:09.046600 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974d93f1-af4a-4829-905d-6921e233746b None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:09.047157 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974d93f1-af4a-4829-905d-6921e233746b None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:09.047589 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974d93f1-af4a-4829-905d-6921e233746b None admin] PATH_INFO: `/v3/domains/default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:09.048924 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974d93f1-af4a-4829-905d-6921e233746b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:09.050843 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974d93f1-af4a-4829-905d-6921e233746b None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:09.052722 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 22/44] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:30:08 2022] GET /identity/v3/domains/default => generated 199 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:09.060061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974d93f1-af4a-4829-905d-6921e233746b None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:09.099889 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:09.155083 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:09.173205 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d97c70-c7cf-4904-a156-c06f84198f82 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:09.200830 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974d93f1-af4a-4829-905d-6921e233746b None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:09.209434 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3bffc59-60db-40da-b1f2-f8ee04403f82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:09.210065 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bffc59-60db-40da-b1f2-f8ee04403f82 None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:09.210342 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bffc59-60db-40da-b1f2-f8ee04403f82 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:09.210659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bffc59-60db-40da-b1f2-f8ee04403f82 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:09.211357 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3bffc59-60db-40da-b1f2-f8ee04403f82 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:09.213312 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3bffc59-60db-40da-b1f2-f8ee04403f82 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:09.238485 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 23/45] 10.222.0.25 () {60 vars in 1131 bytes} [Fri Oct 28 09:30:09 2022] POST /identity/v3/roles => generated 223 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:30:09.239779 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da2bc7ff-3bba-413e-843f-b94d74ccc9ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:09.240428 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2bc7ff-3bba-413e-843f-b94d74ccc9ea None admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:09.240717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2bc7ff-3bba-413e-843f-b94d74ccc9ea None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:09.241002 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da2bc7ff-3bba-413e-843f-b94d74ccc9ea None admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:09.241683 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da2bc7ff-3bba-413e-843f-b94d74ccc9ea None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:09.243322 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da2bc7ff-3bba-413e-843f-b94d74ccc9ea None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:09.281923 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 23/46] 10.222.0.25 () {60 vars in 1141 bytes} [Fri Oct 28 09:30:09 2022] POST /identity/v3/projects => generated 312 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:30:11.024697 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9857ac66-b150-448d-bf95-6a2c336bbbf4 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:11.025238 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9857ac66-b150-448d-bf95-6a2c336bbbf4 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:11.025238 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9857ac66-b150-448d-bf95-6a2c336bbbf4 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:11.026608 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 24/47] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:30:11.032282 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411d996b-9f54-4ec8-a280-b8bfd0529d8b None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:11.032684 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411d996b-9f54-4ec8-a280-b8bfd0529d8b None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:11.032829 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411d996b-9f54-4ec8-a280-b8bfd0529d8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:11.052795 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-411d996b-9f54-4ec8-a280-b8bfd0529d8b None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:11.054298 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-411d996b-9f54-4ec8-a280-b8bfd0529d8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:11.121640 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 24/48] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:30:11.390193 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9857ac66-b150-448d-bf95-6a2c336bbbf4 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:11.474212 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fa20a5-22ae-4547-9ec4-df4230baea2f None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:11.474535 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fa20a5-22ae-4547-9ec4-df4230baea2f None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:11.474813 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fa20a5-22ae-4547-9ec4-df4230baea2f None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:11.476148 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 25/49] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 09:30:11.481637 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245ba2e0-132c-4d61-9a56-406ec1e1b282 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:11.481925 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245ba2e0-132c-4d61-9a56-406ec1e1b282 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:11.482206 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245ba2e0-132c-4d61-9a56-406ec1e1b282 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:11.492464 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9857ac66-b150-448d-bf95-6a2c336bbbf4 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:11.494827 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:11.500498 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-245ba2e0-132c-4d61-9a56-406ec1e1b282 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:11.502663 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-245ba2e0-132c-4d61-9a56-406ec1e1b282 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:11.527925 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1975af-87c2-4601-8926-bad6f36d74c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:11.528914 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1975af-87c2-4601-8926-bad6f36d74c4 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:11.529204 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1975af-87c2-4601-8926-bad6f36d74c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:11.529486 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1975af-87c2-4601-8926-bad6f36d74c4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:11.540397 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e1975af-87c2-4601-8926-bad6f36d74c4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:30:11.545340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-2e1975af-87c2-4601-8926-bad6f36d74c4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:30:11.545340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:30:11.545340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:30:11.545340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:30:11.545340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:30:11.545340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:30:11.545340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:30:11.545340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:30:11.545340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:30:11.545340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:30:11.545340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:30:11.545340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:30:11.545340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:30:11.545340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:30:11.545340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:30:11.545340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:30:11.546375 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:30:11.546375 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:30:11.546375 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:30:11.546375 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:30:11.546375 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:30:11.546375 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:30:11.546375 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:30:11.546375 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:30:11.546375 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:30:11.546375 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:30:11.546375 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:30:11.546375 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:30:11.546375 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:30:11.546375 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:30:11.546375 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:30:11.546375 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:30:11.547172 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:30:11.547172 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:30:11.547172 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:30:11.547172 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:30:11.547172 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:30:11.547172 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:30:11.547172 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:30:11.547172 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:30:11.547172 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:30:11.547172 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:30:11.547172 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:30:11.547172 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:30:11.547172 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 25/50] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Oct 28 09:30:11 2022] GET /identity/v3/roles/admin => generated 83 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:30:11.552437 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1975af-87c2-4601-8926-bad6f36d74c4 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:11.561906 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:11.576990 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 26/51] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 09:30:11.657517 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1975af-87c2-4601-8926-bad6f36d74c4 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:11.794812 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4a19a0-0a56-4fa1-a3bb-3d7d06a0c5c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:11.795429 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4a19a0-0a56-4fa1-a3bb-3d7d06a0c5c0 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:11.795718 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4a19a0-0a56-4fa1-a3bb-3d7d06a0c5c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:11.796002 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4a19a0-0a56-4fa1-a3bb-3d7d06a0c5c0 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:11.796673 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be4a19a0-0a56-4fa1-a3bb-3d7d06a0c5c0 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:11.796983 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be4a19a0-0a56-4fa1-a3bb-3d7d06a0c5c0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:11.798348 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be4a19a0-0a56-4fa1-a3bb-3d7d06a0c5c0 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:11.806599 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 26/52] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:30:11 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:30:11.812435 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245ba2e0-132c-4d61-9a56-406ec1e1b282 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:11.845024 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4a19a0-0a56-4fa1-a3bb-3d7d06a0c5c0 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:11.914058 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245ba2e0-132c-4d61-9a56-406ec1e1b282 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:11.915867 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:11.950305 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be25943d-9783-4b80-810b-6ebbe1156425 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:11.950982 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be25943d-9783-4b80-810b-6ebbe1156425 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:11.951269 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be25943d-9783-4b80-810b-6ebbe1156425 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:11.951423 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4a19a0-0a56-4fa1-a3bb-3d7d06a0c5c0 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:11.951545 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be25943d-9783-4b80-810b-6ebbe1156425 None admin] PATH_INFO: `/v3/users/8afbf6a033494d4cbbdea7a924ab5c63` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:11.952671 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be25943d-9783-4b80-810b-6ebbe1156425 None admin] RBAC: Authorizing `identity:get_user(user_id=8afbf6a033494d4cbbdea7a924ab5c63)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:11.954132 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be25943d-9783-4b80-810b-6ebbe1156425 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:11.955750 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 27/53] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:30:11 2022] GET /identity/v3/users/8afbf6a033494d4cbbdea7a924ab5c63 => generated 247 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:11.961095 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be25943d-9783-4b80-810b-6ebbe1156425 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:11.964714 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:11.985407 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337e37ee-5489-4a30-9dda-1f76f915aae5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:11.985943 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337e37ee-5489-4a30-9dda-1f76f915aae5 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:11.986232 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337e37ee-5489-4a30-9dda-1f76f915aae5 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:11.986531 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337e37ee-5489-4a30-9dda-1f76f915aae5 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:11.987637 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-337e37ee-5489-4a30-9dda-1f76f915aae5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:11.989069 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-337e37ee-5489-4a30-9dda-1f76f915aae5 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:11.990675 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 27/54] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:30:11 2022] GET /identity/v3/domains/default => generated 199 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:30:11.996590 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337e37ee-5489-4a30-9dda-1f76f915aae5 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:12.063656 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be25943d-9783-4b80-810b-6ebbe1156425 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:12.096623 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e3319c-034b-4e7e-a889-e6d809dd581e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:12.097275 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e3319c-034b-4e7e-a889-e6d809dd581e None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:12.097558 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e3319c-034b-4e7e-a889-e6d809dd581e None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:12.097831 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e3319c-034b-4e7e-a889-e6d809dd581e None admin] PATH_INFO: `/v3/domains/default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:12.099063 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e3319c-034b-4e7e-a889-e6d809dd581e None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:12.100569 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e3319c-034b-4e7e-a889-e6d809dd581e None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:12.101124 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337e37ee-5489-4a30-9dda-1f76f915aae5 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:12.102290 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 28/55] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:30:11 2022] GET /identity/v3/domains/default => generated 199 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:30:12.108304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e3319c-034b-4e7e-a889-e6d809dd581e None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:12.137265 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0672a038-6dbe-4718-bad3-b4e357b7f582 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:12.137884 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0672a038-6dbe-4718-bad3-b4e357b7f582 None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:12.138170 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0672a038-6dbe-4718-bad3-b4e357b7f582 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:12.138482 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0672a038-6dbe-4718-bad3-b4e357b7f582 None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:12.139140 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0672a038-6dbe-4718-bad3-b4e357b7f582 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:12.140572 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0672a038-6dbe-4718-bad3-b4e357b7f582 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:12.151689 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:12.172076 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 28/56] 10.222.0.25 () {60 vars in 1140 bytes} [Fri Oct 28 09:30:11 2022] POST /identity/v3/projects => generated 298 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:30:12.184880 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:12.217982 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e3319c-034b-4e7e-a889-e6d809dd581e None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:12.255126 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-220a3511-a7a8-4e47-b08c-79c4a2efbbe7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:12.255772 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220a3511-a7a8-4e47-b08c-79c4a2efbbe7 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:12.256074 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220a3511-a7a8-4e47-b08c-79c4a2efbbe7 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:12.256350 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220a3511-a7a8-4e47-b08c-79c4a2efbbe7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:12.257093 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-220a3511-a7a8-4e47-b08c-79c4a2efbbe7 None admin] RBAC: Adding query filter params (role.id=e13d798a98ea4c3dbb87a236ec683d32, scope.domain.id=default, user.id=8afbf6a033494d4cbbdea7a924ab5c63) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:12.257440 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-220a3511-a7a8-4e47-b08c-79c4a2efbbe7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:12.259485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-220a3511-a7a8-4e47-b08c-79c4a2efbbe7 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:12.266051 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 29/57] 10.222.0.25 () {56 vars in 1322 bytes} [Fri Oct 28 09:30:12 2022] GET /identity/v3/role_assignments?user.id=8afbf6a033494d4cbbdea7a924ab5c63&scope.domain.id=default&role.id=e13d798a98ea4c3dbb87a236ec683d32 => generated 530 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:30:14.330793 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7924a63b-e65c-4313-84fa-d32e17f1822d None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:14.331305 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7924a63b-e65c-4313-84fa-d32e17f1822d None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:14.331391 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7924a63b-e65c-4313-84fa-d32e17f1822d None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:14.332718 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 29/58] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:30:14.338556 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88fb7c53-8d4f-428b-ba6b-e34ee5896d07 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:14.338973 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88fb7c53-8d4f-428b-ba6b-e34ee5896d07 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:14.339121 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88fb7c53-8d4f-428b-ba6b-e34ee5896d07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:14.359322 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-88fb7c53-8d4f-428b-ba6b-e34ee5896d07 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:14.360517 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-88fb7c53-8d4f-428b-ba6b-e34ee5896d07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:14.428622 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 30/59] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 09:30:14.698620 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7924a63b-e65c-4313-84fa-d32e17f1822d None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:14.801224 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7924a63b-e65c-4313-84fa-d32e17f1822d None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:14.834413 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8340dda2-4c67-4fd8-9a12-3b562fc9d6e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:14.835260 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8340dda2-4c67-4fd8-9a12-3b562fc9d6e3 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:14.835691 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8340dda2-4c67-4fd8-9a12-3b562fc9d6e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:14.835968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8340dda2-4c67-4fd8-9a12-3b562fc9d6e3 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:14.837627 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8340dda2-4c67-4fd8-9a12-3b562fc9d6e3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:14.839112 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8340dda2-4c67-4fd8-9a12-3b562fc9d6e3 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:14.840659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 30/60] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:30:14 2022] GET /identity/v3/domains/default => generated 199 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:14.846252 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88fb7c53-8d4f-428b-ba6b-e34ee5896d07 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:14.911195 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:14.946546 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88fb7c53-8d4f-428b-ba6b-e34ee5896d07 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:14.980480 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c575a8b-3e6c-4c65-b3b5-1cf4ee33c93e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:14.981022 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c575a8b-3e6c-4c65-b3b5-1cf4ee33c93e None admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:14.981306 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c575a8b-3e6c-4c65-b3b5-1cf4ee33c93e None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:14.981581 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c575a8b-3e6c-4c65-b3b5-1cf4ee33c93e None admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:14.982244 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c575a8b-3e6c-4c65-b3b5-1cf4ee33c93e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:14.983723 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c575a8b-3e6c-4c65-b3b5-1cf4ee33c93e None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:15.009354 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 31/61] 10.222.0.25 () {60 vars in 1132 bytes} [Fri Oct 28 09:30:14 2022] POST /identity/v3/users => generated 275 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:30:18.326421 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a00261-6d4a-418d-a9aa-370be5b902f2 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:18.326421 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a00261-6d4a-418d-a9aa-370be5b902f2 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:18.326421 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a00261-6d4a-418d-a9aa-370be5b902f2 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:18.327934 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 31/62] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:30:18.333638 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d428d5-d879-47f7-aaa3-b5751a2c1281 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:18.334489 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d428d5-d879-47f7-aaa3-b5751a2c1281 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:18.334969 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d428d5-d879-47f7-aaa3-b5751a2c1281 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:18.361665 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-b4d428d5-d879-47f7-aaa3-b5751a2c1281 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:18.363213 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-b4d428d5-d879-47f7-aaa3-b5751a2c1281 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:18.408369 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65f9508-b1f5-44ac-9aa6-87eb3f602885 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:18.409122 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65f9508-b1f5-44ac-9aa6-87eb3f602885 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:18.409595 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65f9508-b1f5-44ac-9aa6-87eb3f602885 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:18.411162 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 32/63] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 09:30:18.417544 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad85e702-c54b-4ff4-9a3d-18b1555a072f None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:18.418236 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad85e702-c54b-4ff4-9a3d-18b1555a072f None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:18.418744 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad85e702-c54b-4ff4-9a3d-18b1555a072f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:18.449578 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-ad85e702-c54b-4ff4-9a3d-18b1555a072f None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:18.451315 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ad85e702-c54b-4ff4-9a3d-18b1555a072f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:18.504447 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 32/64] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:30:18.536103 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:18.551044 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 33/65] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 09:30:18.769265 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b13489-60cd-4925-9682-65204f888a27 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:18.769935 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b13489-60cd-4925-9682-65204f888a27 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:18.770431 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b13489-60cd-4925-9682-65204f888a27 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:18.772272 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 33/66] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:30:18.778095 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e5a172-2302-4b18-b4a5-907642267c56 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:18.778676 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e5a172-2302-4b18-b4a5-907642267c56 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:18.779150 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e5a172-2302-4b18-b4a5-907642267c56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:18.811749 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-30e5a172-2302-4b18-b4a5-907642267c56 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:18.813374 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-30e5a172-2302-4b18-b4a5-907642267c56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:18.827954 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78b13489-60cd-4925-9682-65204f888a27 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:18.897704 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:18.955068 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78b13489-60cd-4925-9682-65204f888a27 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:18.960571 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 34/67] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:30:18.961943 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e5a172-2302-4b18-b4a5-907642267c56 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:19.001930 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328d6f47-bbd3-4272-b517-872ed7b32d62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:19.003241 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328d6f47-bbd3-4272-b517-872ed7b32d62 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:19.004667 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328d6f47-bbd3-4272-b517-872ed7b32d62 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:19.005153 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328d6f47-bbd3-4272-b517-872ed7b32d62 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:19.020877 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328d6f47-bbd3-4272-b517-872ed7b32d62 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:30:19.028681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-328d6f47-bbd3-4272-b517-872ed7b32d62 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:30:19.028681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:30:19.028681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:30:19.028681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:30:19.028681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:30:19.028681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:30:19.028681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:30:19.028681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:30:19.028681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:30:19.028681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:30:19.028681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:30:19.028681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:30:19.028681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:30:19.028681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:30:19.028681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:30:19.028681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:30:19.029565 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:30:19.029565 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:30:19.029565 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:30:19.029565 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:30:19.029565 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:30:19.029565 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:30:19.029565 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:30:19.029565 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:30:19.029565 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:30:19.029565 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:30:19.029565 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:30:19.029565 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:30:19.029565 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:30:19.029565 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:30:19.029565 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:30:19.029565 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:30:19.030306 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:30:19.030306 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:30:19.030306 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:30:19.030306 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:30:19.030306 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:30:19.030306 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:30:19.030306 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:30:19.030306 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:30:19.030306 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:30:19.030306 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:30:19.030306 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:30:19.030306 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:30:19.032056 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 34/68] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Oct 28 09:30:18 2022] GET /identity/v3/roles/admin => generated 83 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:30:19.033737 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b1db03-b7b3-4e62-a2f8-9624239f0875 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:19.034025 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b1db03-b7b3-4e62-a2f8-9624239f0875 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:19.034303 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b1db03-b7b3-4e62-a2f8-9624239f0875 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:19.035787 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 35/69] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:30:19.040574 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b1db03-b7b3-4e62-a2f8-9624239f0875 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:19.081096 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e5a172-2302-4b18-b4a5-907642267c56 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:19.108638 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:19.136329 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-364704dc-b9c3-4ea0-b6b2-3f0c78325a55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:19.137893 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364704dc-b9c3-4ea0-b6b2-3f0c78325a55 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:19.138671 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364704dc-b9c3-4ea0-b6b2-3f0c78325a55 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:19.139141 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364704dc-b9c3-4ea0-b6b2-3f0c78325a55 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:19.154312 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-364704dc-b9c3-4ea0-b6b2-3f0c78325a55 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:30:19.161380 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-364704dc-b9c3-4ea0-b6b2-3f0c78325a55 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 09:30:19.161380 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:30:19.161380 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:30:19.161380 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:30:19.161380 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:30:19.161380 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:30:19.161380 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:30:19.161380 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:30:19.161380 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:30:19.161380 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:30:19.161380 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:30:19.161380 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:30:19.161380 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:30:19.161380 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:30:19.161380 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:30:19.161380 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:30:19.162241 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:30:19.162241 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:30:19.162241 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:30:19.162241 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:30:19.162241 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:30:19.162241 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:30:19.162241 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:30:19.162241 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:30:19.162241 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:30:19.162241 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:30:19.162241 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:30:19.162241 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:30:19.162241 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:30:19.162241 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:30:19.162241 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:30:19.162241 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:30:19.163011 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:30:19.163011 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:30:19.163011 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:30:19.163011 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:30:19.163011 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:30:19.163011 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:30:19.163011 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:30:19.163011 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:30:19.163011 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:30:19.163011 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:30:19.163011 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 09:30:19.163011 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:30:19.164485 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 35/70] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:30:18 2022] GET /identity/v3/roles/member => generated 84 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:30:19.166457 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a141065-4520-4e00-a9a2-0e374ed9c69f None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:19.166747 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a141065-4520-4e00-a9a2-0e374ed9c69f None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:19.167027 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a141065-4520-4e00-a9a2-0e374ed9c69f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:19.180792 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b1db03-b7b3-4e62-a2f8-9624239f0875 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:19.185444 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-3a141065-4520-4e00-a9a2-0e374ed9c69f None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:19.186674 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-3a141065-4520-4e00-a9a2-0e374ed9c69f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:19.199968 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:19.225622 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249ed423-1e60-4461-bee9-927da899bbf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:19.226242 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249ed423-1e60-4461-bee9-927da899bbf2 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:19.226586 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249ed423-1e60-4461-bee9-927da899bbf2 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:19.226860 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249ed423-1e60-4461-bee9-927da899bbf2 None admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:19.228058 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249ed423-1e60-4461-bee9-927da899bbf2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:19.228366 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249ed423-1e60-4461-bee9-927da899bbf2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:19.229767 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249ed423-1e60-4461-bee9-927da899bbf2 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:19.238429 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 36/71] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:30:19 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:30:19.239133 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249ed423-1e60-4461-bee9-927da899bbf2 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:19.265700 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 36/72] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:30:19.267216 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3ea70b-f2cb-48f7-9355-9f20b0fdb0e4 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:19.267517 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3ea70b-f2cb-48f7-9355-9f20b0fdb0e4 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:19.267799 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a3ea70b-f2cb-48f7-9355-9f20b0fdb0e4 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:19.269198 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 37/73] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:30:19.273976 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a3ea70b-f2cb-48f7-9355-9f20b0fdb0e4 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:19.351221 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249ed423-1e60-4461-bee9-927da899bbf2 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:19.389325 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b7d4ed-c115-4b83-b150-a29dbd4a67e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:19.389943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b7d4ed-c115-4b83-b150-a29dbd4a67e5 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:19.390229 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b7d4ed-c115-4b83-b150-a29dbd4a67e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:19.390538 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18b7d4ed-c115-4b83-b150-a29dbd4a67e5 None admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:19.390849 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a3ea70b-f2cb-48f7-9355-9f20b0fdb0e4 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:19.391266 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18b7d4ed-c115-4b83-b150-a29dbd4a67e5 None admin] RBAC: Adding query filter params (name=member) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:19.391503 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18b7d4ed-c115-4b83-b150-a29dbd4a67e5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:19.392906 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18b7d4ed-c115-4b83-b150-a29dbd4a67e5 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:19.399237 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:19.401910 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 37/74] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Oct 28 09:30:19 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:19.402362 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b7d4ed-c115-4b83-b150-a29dbd4a67e5 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:19.441594 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:19.441839 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c431620a-f2a6-433f-b71e-4f7296275c32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:19.442725 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c431620a-f2a6-433f-b71e-4f7296275c32 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:19.443214 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c431620a-f2a6-433f-b71e-4f7296275c32 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:19.443681 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c431620a-f2a6-433f-b71e-4f7296275c32 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:19.455542 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c431620a-f2a6-433f-b71e-4f7296275c32 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:30:19.460425 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-c431620a-f2a6-433f-b71e-4f7296275c32 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 09:30:19.460425 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:30:19.460425 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:30:19.460425 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:30:19.460425 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:30:19.460425 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:30:19.460425 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:30:19.460425 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:30:19.460425 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:30:19.460425 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:30:19.460425 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:30:19.460425 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:30:19.460425 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:30:19.460425 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:30:19.460425 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:30:19.460425 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:30:19.461261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:30:19.461261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:30:19.461261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:30:19.461261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:30:19.461261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:30:19.461261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:30:19.461261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:30:19.461261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:30:19.461261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:30:19.461261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:30:19.461261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:30:19.461261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:30:19.461261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:30:19.461261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:30:19.461261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:30:19.461261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:30:19.462622 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:30:19.462622 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:30:19.462622 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:30:19.462622 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:30:19.462622 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:30:19.462622 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:30:19.462622 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:30:19.462622 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:30:19.462622 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:30:19.462622 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:30:19.462622 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 09:30:19.462622 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:30:19.462622 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 38/75] 10.222.0.25 () {56 vars in 1114 bytes} [Fri Oct 28 09:30:19 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:30:19.463899 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3250dfd5-5b24-4d11-8893-54c6f1a2473a None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:19.463899 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3250dfd5-5b24-4d11-8893-54c6f1a2473a None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:19.464059 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3250dfd5-5b24-4d11-8893-54c6f1a2473a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:19.486451 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-3250dfd5-5b24-4d11-8893-54c6f1a2473a None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:19.487657 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-3250dfd5-5b24-4d11-8893-54c6f1a2473a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:19.525360 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18b7d4ed-c115-4b83-b150-a29dbd4a67e5 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:19.561200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d51092-e59d-4862-a3f6-84dcef5eca85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:19.561843 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d51092-e59d-4862-a3f6-84dcef5eca85 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:19.562127 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d51092-e59d-4862-a3f6-84dcef5eca85 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:19.562424 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d51092-e59d-4862-a3f6-84dcef5eca85 None admin] PATH_INFO: `/v3/users/8afbf6a033494d4cbbdea7a924ab5c63` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:19.563567 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7d51092-e59d-4862-a3f6-84dcef5eca85 None admin] RBAC: Authorizing `identity:get_user(user_id=8afbf6a033494d4cbbdea7a924ab5c63)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:19.564835 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 39/76] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:30:19.565039 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7d51092-e59d-4862-a3f6-84dcef5eca85 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:19.565693 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3250dfd5-5b24-4d11-8893-54c6f1a2473a None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:19.574329 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 38/77] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:30:19 2022] GET /identity/v3/users/8afbf6a033494d4cbbdea7a924ab5c63 => generated 247 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:19.574329 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d51092-e59d-4862-a3f6-84dcef5eca85 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:19.673459 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:19.681947 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3250dfd5-5b24-4d11-8893-54c6f1a2473a None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:19.687966 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d51092-e59d-4862-a3f6-84dcef5eca85 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:19.691128 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:19.718307 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b44a0a3-9f88-4b84-abce-0eec20d4fa84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:19.718905 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b44a0a3-9f88-4b84-abce-0eec20d4fa84 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:19.719168 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b44a0a3-9f88-4b84-abce-0eec20d4fa84 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:19.719449 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b44a0a3-9f88-4b84-abce-0eec20d4fa84 None admin] PATH_INFO: `/v3/users/c717178de72f493788554ab410b527de` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:19.726656 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90fcc0b4-aa22-49bc-952b-c89113584133 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:19.727333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fcc0b4-aa22-49bc-952b-c89113584133 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:19.727720 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fcc0b4-aa22-49bc-952b-c89113584133 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:19.728006 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90fcc0b4-aa22-49bc-952b-c89113584133 None admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:19.728825 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90fcc0b4-aa22-49bc-952b-c89113584133 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:19.729372 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90fcc0b4-aa22-49bc-952b-c89113584133 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:19.730818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90fcc0b4-aa22-49bc-952b-c89113584133 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:19.738622 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b44a0a3-9f88-4b84-abce-0eec20d4fa84 None admin] RBAC: Authorizing `identity:get_user(user_id=c717178de72f493788554ab410b527de)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:19.739487 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 39/78] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:30:19 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:19.740118 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90fcc0b4-aa22-49bc-952b-c89113584133 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:19.745258 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b44a0a3-9f88-4b84-abce-0eec20d4fa84 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:19.750538 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 40/79] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:30:19 2022] GET /identity/v3/users/c717178de72f493788554ab410b527de => generated 275 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:19.750538 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b44a0a3-9f88-4b84-abce-0eec20d4fa84 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:19.849789 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90fcc0b4-aa22-49bc-952b-c89113584133 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:19.860348 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b44a0a3-9f88-4b84-abce-0eec20d4fa84 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:19.884553 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da4adcec-19ff-4c21-b1fb-e2cc9973b79a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:19.885184 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4adcec-19ff-4c21-b1fb-e2cc9973b79a None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:19.885464 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4adcec-19ff-4c21-b1fb-e2cc9973b79a None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:19.885739 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4adcec-19ff-4c21-b1fb-e2cc9973b79a None admin] PATH_INFO: `/v3/roles/member` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:19.894201 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07106430-0812-4d64-91c2-493e8f73027e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:19.894843 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07106430-0812-4d64-91c2-493e8f73027e None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:19.895131 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07106430-0812-4d64-91c2-493e8f73027e None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:19.895410 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07106430-0812-4d64-91c2-493e8f73027e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:19.896076 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da4adcec-19ff-4c21-b1fb-e2cc9973b79a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:30:19.901373 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-da4adcec-19ff-4c21-b1fb-e2cc9973b79a None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 09:30:19.901373 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:30:19.901373 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:30:19.901373 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:30:19.901373 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:30:19.901373 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:30:19.901373 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:30:19.901373 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:30:19.901373 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:30:19.901373 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:30:19.901373 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:30:19.901373 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:30:19.901373 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:30:19.901373 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:30:19.901373 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:30:19.901373 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:30:19.902334 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:30:19.902334 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:30:19.902334 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:30:19.902334 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:30:19.902334 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:30:19.902334 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:30:19.902334 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:30:19.902334 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:30:19.902334 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:30:19.902334 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:30:19.902334 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:30:19.902334 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:30:19.902334 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:30:19.902334 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:30:19.902334 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:30:19.902334 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:30:19.903116 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:30:19.903116 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:30:19.903116 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:30:19.903116 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:30:19.903116 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:30:19.903116 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:30:19.903116 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:30:19.903116 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:30:19.903116 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:30:19.903116 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:30:19.903116 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 09:30:19.903116 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:30:19.903116 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 40/80] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:30:19 2022] GET /identity/v3/roles/member => generated 84 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:30:19.903737 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da4adcec-19ff-4c21-b1fb-e2cc9973b79a None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:19.921979 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:19.932320 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:19.973562 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07106430-0812-4d64-91c2-493e8f73027e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:19.975471 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07106430-0812-4d64-91c2-493e8f73027e None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:20.013080 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da4adcec-19ff-4c21-b1fb-e2cc9973b79a None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:20.053427 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-492ed6c3-dbef-4c2d-86e4-383909a6306b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:20.054203 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492ed6c3-dbef-4c2d-86e4-383909a6306b None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:20.054585 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492ed6c3-dbef-4c2d-86e4-383909a6306b None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:20.054961 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492ed6c3-dbef-4c2d-86e4-383909a6306b None admin] PATH_INFO: `/v3/users/c717178de72f493788554ab410b527de` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:20.058071 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-492ed6c3-dbef-4c2d-86e4-383909a6306b None admin] RBAC: Authorizing `identity:get_user(user_id=c717178de72f493788554ab410b527de)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:20.059590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-492ed6c3-dbef-4c2d-86e4-383909a6306b None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:20.061218 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 41/81] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:30:19 2022] GET /identity/v3/users/c717178de72f493788554ab410b527de => generated 275 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:20.062001 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-492ed6c3-dbef-4c2d-86e4-383909a6306b None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:20.093811 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 41/82] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:30:19 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 09:30:20.094712 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07106430-0812-4d64-91c2-493e8f73027e None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:20.153804 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:20.160664 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:20.168893 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-492ed6c3-dbef-4c2d-86e4-383909a6306b None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:20.201745 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07106430-0812-4d64-91c2-493e8f73027e None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:20.203881 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e77547e-50ab-4f9f-8d5c-9faac6649fa7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:20.204500 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e77547e-50ab-4f9f-8d5c-9faac6649fa7 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:20.204812 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e77547e-50ab-4f9f-8d5c-9faac6649fa7 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:20.205086 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e77547e-50ab-4f9f-8d5c-9faac6649fa7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:20.241433 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85f4add1-2e1b-4742-b4de-f04184d49fab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:20.241995 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f4add1-2e1b-4742-b4de-f04184d49fab None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:20.242283 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f4add1-2e1b-4742-b4de-f04184d49fab None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:20.243111 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f4add1-2e1b-4742-b4de-f04184d49fab None admin] PATH_INFO: `/v3/domains/default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:20.244298 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85f4add1-2e1b-4742-b4de-f04184d49fab None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:20.245746 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85f4add1-2e1b-4742-b4de-f04184d49fab None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:20.247365 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 42/83] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:30:20 2022] GET /identity/v3/domains/default => generated 199 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:20.252149 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85f4add1-2e1b-4742-b4de-f04184d49fab None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:20.292538 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e77547e-50ab-4f9f-8d5c-9faac6649fa7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:20.294644 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e77547e-50ab-4f9f-8d5c-9faac6649fa7 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:20.355775 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85f4add1-2e1b-4742-b4de-f04184d49fab None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:20.381039 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:20.389243 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:20.393765 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3220ac11-7450-4b35-91e8-11e316a08df4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:20.394379 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3220ac11-7450-4b35-91e8-11e316a08df4 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:20.394693 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3220ac11-7450-4b35-91e8-11e316a08df4 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:20.394970 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3220ac11-7450-4b35-91e8-11e316a08df4 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:20.395634 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3220ac11-7450-4b35-91e8-11e316a08df4 None admin] RBAC: Adding query filter params (name=member) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:20.395942 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3220ac11-7450-4b35-91e8-11e316a08df4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:20.396898 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 42/84] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:30:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 335 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 09:30:20.397317 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3220ac11-7450-4b35-91e8-11e316a08df4 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:20.397713 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e77547e-50ab-4f9f-8d5c-9faac6649fa7 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:20.405809 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 43/85] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Oct 28 09:30:20 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:20.407125 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3220ac11-7450-4b35-91e8-11e316a08df4 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:20.500615 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e77547e-50ab-4f9f-8d5c-9faac6649fa7 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:20.506427 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3220ac11-7450-4b35-91e8-11e316a08df4 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:20.534739 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef967f03-003d-4231-afa1-4ce88c6d33d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:20.535357 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef967f03-003d-4231-afa1-4ce88c6d33d0 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:20.535642 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef967f03-003d-4231-afa1-4ce88c6d33d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:20.535926 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef967f03-003d-4231-afa1-4ce88c6d33d0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:20.541923 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5015b28-7f2c-4087-bb1e-d969d06f3f79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:20.542591 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5015b28-7f2c-4087-bb1e-d969d06f3f79 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:20.542922 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5015b28-7f2c-4087-bb1e-d969d06f3f79 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:20.543201 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5015b28-7f2c-4087-bb1e-d969d06f3f79 None admin] PATH_INFO: `/v3/projects/6e28083c55ff4290af640f011fd2466f` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:20.544342 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5015b28-7f2c-4087-bb1e-d969d06f3f79 None admin] RBAC: Authorizing `identity:get_project(project_id=6e28083c55ff4290af640f011fd2466f)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:20.545778 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5015b28-7f2c-4087-bb1e-d969d06f3f79 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:20.547485 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 44/86] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:30:20 2022] GET /identity/v3/projects/6e28083c55ff4290af640f011fd2466f => generated 298 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:20.548269 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5015b28-7f2c-4087-bb1e-d969d06f3f79 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:20.602701 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:20.605890 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:20.613669 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef967f03-003d-4231-afa1-4ce88c6d33d0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:20.615196 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef967f03-003d-4231-afa1-4ce88c6d33d0 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:20.658864 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5015b28-7f2c-4087-bb1e-d969d06f3f79 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:20.694696 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3aa361c-57e2-415f-bcaf-ae293a8237b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:20.695411 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3aa361c-57e2-415f-bcaf-ae293a8237b7 None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:20.696293 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3aa361c-57e2-415f-bcaf-ae293a8237b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:20.696691 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3aa361c-57e2-415f-bcaf-ae293a8237b7 None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:20.697346 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3aa361c-57e2-415f-bcaf-ae293a8237b7 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:20.698741 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3aa361c-57e2-415f-bcaf-ae293a8237b7 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:20.718018 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 43/87] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:30:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 09:30:20.718018 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef967f03-003d-4231-afa1-4ce88c6d33d0 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:20.726369 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 45/88] 10.222.0.25 () {60 vars in 1132 bytes} [Fri Oct 28 09:30:20 2022] POST /identity/v3/users => generated 289 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:30:20.727494 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3aa361c-57e2-415f-bcaf-ae293a8237b7 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:20.822693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:20.829885 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef967f03-003d-4231-afa1-4ce88c6d33d0 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:20.862767 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3aa361c-57e2-415f-bcaf-ae293a8237b7 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:20.867238 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee949874-0330-4bc5-af1c-2fbdfcaaf85c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:20.867931 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee949874-0330-4bc5-af1c-2fbdfcaaf85c None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:20.868357 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee949874-0330-4bc5-af1c-2fbdfcaaf85c None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:20.868642 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee949874-0330-4bc5-af1c-2fbdfcaaf85c None admin] PATH_INFO: `/v3/projects/6e28083c55ff4290af640f011fd2466f` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:20.870089 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee949874-0330-4bc5-af1c-2fbdfcaaf85c None admin] RBAC: Authorizing `identity:get_project(project_id=6e28083c55ff4290af640f011fd2466f)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:20.871639 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee949874-0330-4bc5-af1c-2fbdfcaaf85c None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:20.873348 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 44/89] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:30:20 2022] GET /identity/v3/projects/6e28083c55ff4290af640f011fd2466f => generated 298 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:20.873348 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee949874-0330-4bc5-af1c-2fbdfcaaf85c None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:20.880690 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:20.913983 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217e7081-059c-410b-b663-658ec36d985d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:20.914661 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217e7081-059c-410b-b663-658ec36d985d None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:20.914948 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217e7081-059c-410b-b663-658ec36d985d None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:20.915229 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217e7081-059c-410b-b663-658ec36d985d None admin] PATH_INFO: `/v3/users/c717178de72f493788554ab410b527de` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:20.916392 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-217e7081-059c-410b-b663-658ec36d985d None admin] RBAC: Authorizing `identity:get_user(user_id=c717178de72f493788554ab410b527de)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:20.917852 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-217e7081-059c-410b-b663-658ec36d985d None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:20.919786 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 46/90] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:30:20 2022] GET /identity/v3/users/c717178de72f493788554ab410b527de => generated 275 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:20.920268 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217e7081-059c-410b-b663-658ec36d985d None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:21.003023 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee949874-0330-4bc5-af1c-2fbdfcaaf85c None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:21.047256 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce1f0553-9d3a-428f-a89a-e6a4ae888316 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:21.048016 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1f0553-9d3a-428f-a89a-e6a4ae888316 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:21.048305 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1f0553-9d3a-428f-a89a-e6a4ae888316 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:21.048584 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1f0553-9d3a-428f-a89a-e6a4ae888316 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:21.049507 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce1f0553-9d3a-428f-a89a-e6a4ae888316 None admin] RBAC: Adding query filter params (role.id=e13d798a98ea4c3dbb87a236ec683d32, scope.project.id=6e28083c55ff4290af640f011fd2466f, user.id=8afbf6a033494d4cbbdea7a924ab5c63) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:21.049827 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce1f0553-9d3a-428f-a89a-e6a4ae888316 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:21.052133 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce1f0553-9d3a-428f-a89a-e6a4ae888316 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:21.059686 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 45/91] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:30:20 2022] GET /identity/v3/role_assignments?user.id=8afbf6a033494d4cbbdea7a924ab5c63&scope.project.id=6e28083c55ff4290af640f011fd2466f&role.id=e13d798a98ea4c3dbb87a236ec683d32 => generated 259 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:21.061159 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce1f0553-9d3a-428f-a89a-e6a4ae888316 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:21.065033 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217e7081-059c-410b-b663-658ec36d985d None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:21.111925 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f519ef1f-07ac-4ac7-9de8-31f382de7a41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:21.112617 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f519ef1f-07ac-4ac7-9de8-31f382de7a41 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:21.112899 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f519ef1f-07ac-4ac7-9de8-31f382de7a41 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:21.113178 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f519ef1f-07ac-4ac7-9de8-31f382de7a41 None admin] PATH_INFO: `/v3/projects/6e28083c55ff4290af640f011fd2466f` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:21.113977 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:21.114367 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f519ef1f-07ac-4ac7-9de8-31f382de7a41 None admin] RBAC: Authorizing `identity:get_project(project_id=6e28083c55ff4290af640f011fd2466f)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:21.116192 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f519ef1f-07ac-4ac7-9de8-31f382de7a41 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:21.120029 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 47/92] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:30:20 2022] GET /identity/v3/projects/6e28083c55ff4290af640f011fd2466f => generated 298 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:21.120771 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f519ef1f-07ac-4ac7-9de8-31f382de7a41 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:21.179184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:21.286851 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f519ef1f-07ac-4ac7-9de8-31f382de7a41 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:21.292635 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce1f0553-9d3a-428f-a89a-e6a4ae888316 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:21.363983 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa31788-d190-4175-9524-576728f272d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:21.364536 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa31788-d190-4175-9524-576728f272d6 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:21.364826 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa31788-d190-4175-9524-576728f272d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:21.365105 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa31788-d190-4175-9524-576728f272d6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:21.378600 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889ee307-f046-4a55-a688-c1b2956ec41e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:21.379230 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889ee307-f046-4a55-a688-c1b2956ec41e None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:21.379523 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889ee307-f046-4a55-a688-c1b2956ec41e None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:21.379804 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889ee307-f046-4a55-a688-c1b2956ec41e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:21.380554 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-889ee307-f046-4a55-a688-c1b2956ec41e None admin] RBAC: Adding query filter params (role.id=8943f289c9fe4feba2695b9c90cf3d51, scope.project.id=6e28083c55ff4290af640f011fd2466f, user.id=c717178de72f493788554ab410b527de) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:21.380870 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-889ee307-f046-4a55-a688-c1b2956ec41e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:21.382347 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-889ee307-f046-4a55-a688-c1b2956ec41e None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:21.393664 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 46/93] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:30:21 2022] GET /identity/v3/role_assignments?user.id=c717178de72f493788554ab410b527de&scope.project.id=6e28083c55ff4290af640f011fd2466f&role.id=8943f289c9fe4feba2695b9c90cf3d51 => generated 259 bytes in 333 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:21.394533 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889ee307-f046-4a55-a688-c1b2956ec41e None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:21.465148 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aa31788-d190-4175-9524-576728f272d6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:21.470436 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aa31788-d190-4175-9524-576728f272d6 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:21.554808 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:21.668192 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:21.693053 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 48/94] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:30:21 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 572 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 09:30:21.698435 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889ee307-f046-4a55-a688-c1b2956ec41e None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:21.705942 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa31788-d190-4175-9524-576728f272d6 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:21.770441 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f020746d-7d8b-4b2a-a1d0-b2aa0a3f5019 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:21.771070 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f020746d-7d8b-4b2a-a1d0-b2aa0a3f5019 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:21.771352 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f020746d-7d8b-4b2a-a1d0-b2aa0a3f5019 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:21.771627 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f020746d-7d8b-4b2a-a1d0-b2aa0a3f5019 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:21.772365 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f020746d-7d8b-4b2a-a1d0-b2aa0a3f5019 None admin] RBAC: Adding query filter params (role.id=d93b4c94b590457199c3d301f947b0fb, scope.project.id=6e28083c55ff4290af640f011fd2466f, user.id=c717178de72f493788554ab410b527de) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:21.772676 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f020746d-7d8b-4b2a-a1d0-b2aa0a3f5019 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:21.775481 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f020746d-7d8b-4b2a-a1d0-b2aa0a3f5019 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:21.791195 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 47/95] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:30:21 2022] GET /identity/v3/role_assignments?user.id=c717178de72f493788554ab410b527de&scope.project.id=6e28083c55ff4290af640f011fd2466f&role.id=d93b4c94b590457199c3d301f947b0fb => generated 259 bytes in 397 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:21.947009 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa31788-d190-4175-9524-576728f272d6 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:22.013457 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c7b761-e8c2-4c04-8313-6a2c249ac2a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:22.014103 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c7b761-e8c2-4c04-8313-6a2c249ac2a2 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:22.014385 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c7b761-e8c2-4c04-8313-6a2c249ac2a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:22.014686 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c7b761-e8c2-4c04-8313-6a2c249ac2a2 None admin] PATH_INFO: `/v3/projects/f075fea87a3c4191a9ad2ed506c8f6a3` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:22.015827 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8c7b761-e8c2-4c04-8313-6a2c249ac2a2 None admin] RBAC: Authorizing `identity:get_project(project_id=f075fea87a3c4191a9ad2ed506c8f6a3)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:22.017259 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8c7b761-e8c2-4c04-8313-6a2c249ac2a2 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:22.020178 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 49/96] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:30:21 2022] GET /identity/v3/projects/f075fea87a3c4191a9ad2ed506c8f6a3 => generated 312 bytes in 314 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:22.025638 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f020746d-7d8b-4b2a-a1d0-b2aa0a3f5019 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:22.209808 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f020746d-7d8b-4b2a-a1d0-b2aa0a3f5019 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:22.258781 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf7b1211-212a-44ee-b9aa-71347df85ad0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:22.258781 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7b1211-212a-44ee-b9aa-71347df85ad0 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:22.258781 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7b1211-212a-44ee-b9aa-71347df85ad0 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:22.258781 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf7b1211-212a-44ee-b9aa-71347df85ad0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:22.259168 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf7b1211-212a-44ee-b9aa-71347df85ad0 None admin] RBAC: Adding query filter params (role.id=8943f289c9fe4feba2695b9c90cf3d51, scope.project.id=f075fea87a3c4191a9ad2ed506c8f6a3, user.id=c717178de72f493788554ab410b527de) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:22.259168 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf7b1211-212a-44ee-b9aa-71347df85ad0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:22.259986 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf7b1211-212a-44ee-b9aa-71347df85ad0 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:22.266469 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 48/97] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:30:22 2022] GET /identity/v3/role_assignments?user.id=c717178de72f493788554ab410b527de&scope.project.id=f075fea87a3c4191a9ad2ed506c8f6a3&role.id=8943f289c9fe4feba2695b9c90cf3d51 => generated 259 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:23.857010 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22449c03-52fb-42ca-b26e-82db1cf6c769 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:23.857559 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22449c03-52fb-42ca-b26e-82db1cf6c769 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:23.857829 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22449c03-52fb-42ca-b26e-82db1cf6c769 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:23.859199 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 50/98] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:30:23.864775 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a0c2a0-0396-4a0d-b60e-852eaa4bab46 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:23.865244 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a0c2a0-0396-4a0d-b60e-852eaa4bab46 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:23.865318 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a0c2a0-0396-4a0d-b60e-852eaa4bab46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:23.894904 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-14a0c2a0-0396-4a0d-b60e-852eaa4bab46 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:23.896121 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-14a0c2a0-0396-4a0d-b60e-852eaa4bab46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:23.918796 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:23.941971 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b4eaf7-fb70-4c1c-a42f-5db5344db867 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:23.942627 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b4eaf7-fb70-4c1c-a42f-5db5344db867 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:23.943142 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b4eaf7-fb70-4c1c-a42f-5db5344db867 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:23.944796 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 51/99] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 09:30:23.951093 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5741d3d-638f-4182-91a3-e19dbb6c9949 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:23.951609 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5741d3d-638f-4182-91a3-e19dbb6c9949 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:23.952079 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5741d3d-638f-4182-91a3-e19dbb6c9949 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:23.974626 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-c5741d3d-638f-4182-91a3-e19dbb6c9949 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:23.976271 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-c5741d3d-638f-4182-91a3-e19dbb6c9949 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:23.985453 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:24.009103 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 49/100] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:30:24.054628 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 52/101] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 09:30:24.360256 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e12f67b4-e284-4b37-ad48-efd52486ab4c None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:24.360256 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e12f67b4-e284-4b37-ad48-efd52486ab4c None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:24.360256 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e12f67b4-e284-4b37-ad48-efd52486ab4c None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:24.360256 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 50/102] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:30:24.366016 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fca3e9e-9168-435c-97b7-4d555bc7010d None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:24.366463 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fca3e9e-9168-435c-97b7-4d555bc7010d None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:24.366845 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fca3e9e-9168-435c-97b7-4d555bc7010d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:24.394606 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-2fca3e9e-9168-435c-97b7-4d555bc7010d None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:24.395935 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-2fca3e9e-9168-435c-97b7-4d555bc7010d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:24.455685 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e12f67b4-e284-4b37-ad48-efd52486ab4c None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:24.485261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 53/103] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:30:24.495829 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fca3e9e-9168-435c-97b7-4d555bc7010d None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:24.671570 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:24.729351 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e12f67b4-e284-4b37-ad48-efd52486ab4c None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:24.740661 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fca3e9e-9168-435c-97b7-4d555bc7010d None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:24.803813 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8b6615-13c9-4cae-a799-d9277e5328d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:24.804469 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8b6615-13c9-4cae-a799-d9277e5328d7 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:24.804754 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8b6615-13c9-4cae-a799-d9277e5328d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:24.805059 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8b6615-13c9-4cae-a799-d9277e5328d7 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:24.810966 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1bcac9-8061-4598-9f62-0be5b4c70a34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:24.811615 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1bcac9-8061-4598-9f62-0be5b4c70a34 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:24.811895 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1bcac9-8061-4598-9f62-0be5b4c70a34 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:24.812173 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1bcac9-8061-4598-9f62-0be5b4c70a34 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:24.832093 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe8b6615-13c9-4cae-a799-d9277e5328d7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:30:24.842424 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-fe8b6615-13c9-4cae-a799-d9277e5328d7 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:30:24.842424 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:30:24.842424 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:30:24.842424 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:30:24.842424 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:30:24.842424 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:30:24.842424 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:30:24.842424 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:30:24.842424 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:30:24.842424 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:30:24.842424 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:30:24.842424 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:30:24.842424 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:30:24.842424 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:30:24.842424 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:30:24.842424 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:30:24.843343 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:30:24.843343 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:30:24.843343 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:30:24.843343 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:30:24.843343 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:30:24.843343 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:30:24.843343 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:30:24.843343 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:30:24.843343 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:30:24.843343 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:30:24.843343 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:30:24.843343 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:30:24.843343 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:30:24.843343 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:30:24.843343 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:30:24.843343 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:30:24.844086 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:30:24.844086 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:30:24.844086 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:30:24.844086 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:30:24.844086 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:30:24.844086 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:30:24.844086 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:30:24.844086 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:30:24.844086 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:30:24.844086 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:30:24.844086 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:30:24.844086 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:30:24.844086 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 51/104] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Oct 28 09:30:24 2022] GET /identity/v3/roles/admin => generated 83 bytes in 384 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:30:24.844723 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1bcac9-8061-4598-9f62-0be5b4c70a34 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:30:24.850049 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-ed1bcac9-8061-4598-9f62-0be5b4c70a34 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 09:30:24.850049 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:30:24.850049 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:30:24.850049 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:30:24.850049 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:30:24.850049 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:30:24.850049 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:30:24.850049 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:30:24.850049 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:30:24.850049 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:30:24.850049 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:30:24.850049 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:30:24.850049 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:30:24.850049 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:30:24.850049 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:30:24.850049 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:30:24.850963 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:30:24.850963 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:30:24.850963 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:30:24.850963 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:30:24.850963 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:30:24.850963 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:30:24.850963 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:30:24.850963 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:30:24.850963 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:30:24.850963 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:30:24.850963 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:30:24.850963 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:30:24.850963 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:30:24.850963 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:30:24.850963 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:30:24.850963 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:30:24.852827 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:30:24.852827 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:30:24.852827 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:30:24.852827 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:30:24.852827 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:30:24.852827 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:30:24.852827 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:30:24.852827 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:30:24.852827 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:30:24.852827 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:30:24.852827 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 09:30:24.852827 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:30:24.852827 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 54/105] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:30:24 2022] GET /identity/v3/roles/reader => generated 84 bytes in 356 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:30:24.852827 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1bcac9-8061-4598-9f62-0be5b4c70a34 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:24.854508 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97c5455-8ab3-470f-8b2f-ebac275c000c None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:24.854508 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97c5455-8ab3-470f-8b2f-ebac275c000c None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:24.854765 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97c5455-8ab3-470f-8b2f-ebac275c000c None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:24.856077 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 52/106] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:24 2022] GET /identity => generated 270 bytes in 17 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:30:24.869237 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049879c5-7f57-4074-92c5-de7d7e06fa2c None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:24.870052 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049879c5-7f57-4074-92c5-de7d7e06fa2c None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:24.870052 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049879c5-7f57-4074-92c5-de7d7e06fa2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:24.903082 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-049879c5-7f57-4074-92c5-de7d7e06fa2c None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:24.905490 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-049879c5-7f57-4074-92c5-de7d7e06fa2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:24.927347 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:25.006339 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1bcac9-8061-4598-9f62-0be5b4c70a34 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:25.007375 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 53/107] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:30:25.008502 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049879c5-7f57-4074-92c5-de7d7e06fa2c None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:25.052975 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:25.063487 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c3e6be5-2d78-4bdb-9973-93aea00e1dfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:25.064553 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3e6be5-2d78-4bdb-9973-93aea00e1dfd None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:25.065056 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3e6be5-2d78-4bdb-9973-93aea00e1dfd None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:25.065535 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3e6be5-2d78-4bdb-9973-93aea00e1dfd None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:25.066561 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c3e6be5-2d78-4bdb-9973-93aea00e1dfd None admin] RBAC: Adding query filter params (name=admin) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:25.067182 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c3e6be5-2d78-4bdb-9973-93aea00e1dfd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:25.069718 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c3e6be5-2d78-4bdb-9973-93aea00e1dfd None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:25.082526 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 55/108] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:30:24 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:25.088082 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c3e6be5-2d78-4bdb-9973-93aea00e1dfd None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:25.151929 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049879c5-7f57-4074-92c5-de7d7e06fa2c None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:25.198844 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76cb150-5dc9-45ee-9665-4d859f2b7d73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:25.199414 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76cb150-5dc9-45ee-9665-4d859f2b7d73 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:25.199767 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76cb150-5dc9-45ee-9665-4d859f2b7d73 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:25.200048 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76cb150-5dc9-45ee-9665-4d859f2b7d73 None admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:25.200925 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e76cb150-5dc9-45ee-9665-4d859f2b7d73 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:25.201261 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e76cb150-5dc9-45ee-9665-4d859f2b7d73 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:25.203257 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e76cb150-5dc9-45ee-9665-4d859f2b7d73 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:25.210322 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:25.216708 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 54/109] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Oct 28 09:30:25 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:25.218312 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76cb150-5dc9-45ee-9665-4d859f2b7d73 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:25.224288 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c3e6be5-2d78-4bdb-9973-93aea00e1dfd None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:25.270982 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ef8d59-d2da-4f96-b126-f8bc05c68091 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:25.272024 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ef8d59-d2da-4f96-b126-f8bc05c68091 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:25.272507 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ef8d59-d2da-4f96-b126-f8bc05c68091 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:25.272994 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ef8d59-d2da-4f96-b126-f8bc05c68091 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:25.275083 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7ef8d59-d2da-4f96-b126-f8bc05c68091 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:25.277562 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7ef8d59-d2da-4f96-b126-f8bc05c68091 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:25.280168 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 56/110] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:30:25 2022] GET /identity/v3/domains/default => generated 199 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:25.286420 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32194bad-2b0a-4f84-8d68-6c344a5ffa1e None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:25.286925 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32194bad-2b0a-4f84-8d68-6c344a5ffa1e None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:25.287373 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32194bad-2b0a-4f84-8d68-6c344a5ffa1e None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:25.289353 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 57/111] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:30:25.290311 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32194bad-2b0a-4f84-8d68-6c344a5ffa1e None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:25.342460 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76cb150-5dc9-45ee-9665-4d859f2b7d73 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:25.362690 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:25.390951 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8bf71b4-7408-4421-bb6c-db769c4107f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:25.391745 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8bf71b4-7408-4421-bb6c-db769c4107f4 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:25.392037 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8bf71b4-7408-4421-bb6c-db769c4107f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:25.392324 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8bf71b4-7408-4421-bb6c-db769c4107f4 None admin] PATH_INFO: `/v3/users/8afbf6a033494d4cbbdea7a924ab5c63` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:25.393487 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8bf71b4-7408-4421-bb6c-db769c4107f4 None admin] RBAC: Authorizing `identity:get_user(user_id=8afbf6a033494d4cbbdea7a924ab5c63)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:25.395057 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8bf71b4-7408-4421-bb6c-db769c4107f4 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:25.396826 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 55/112] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:30:25 2022] GET /identity/v3/users/8afbf6a033494d4cbbdea7a924ab5c63 => generated 247 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:25.397513 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8bf71b4-7408-4421-bb6c-db769c4107f4 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:25.417669 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32194bad-2b0a-4f84-8d68-6c344a5ffa1e None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:25.459406 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c3bc9f-99a4-4b55-9f3a-2f76b8445dee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:25.460372 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c3bc9f-99a4-4b55-9f3a-2f76b8445dee None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:25.460880 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c3bc9f-99a4-4b55-9f3a-2f76b8445dee None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:25.461900 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c3bc9f-99a4-4b55-9f3a-2f76b8445dee None admin] PATH_INFO: `/v3/users/d287c1589ecb4e1caa5ee5f36033e00e` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:25.470381 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:25.478080 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c3bc9f-99a4-4b55-9f3a-2f76b8445dee None admin] RBAC: Authorizing `identity:get_user(user_id=d287c1589ecb4e1caa5ee5f36033e00e)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:25.479894 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c3bc9f-99a4-4b55-9f3a-2f76b8445dee None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:25.481979 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 58/113] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:30:25 2022] GET /identity/v3/users/d287c1589ecb4e1caa5ee5f36033e00e => generated 289 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:25.488245 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84934825-ec06-4ca0-a5d6-ecb1853d2c56 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:25.488829 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84934825-ec06-4ca0-a5d6-ecb1853d2c56 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:25.489307 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84934825-ec06-4ca0-a5d6-ecb1853d2c56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:25.508421 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-84934825-ec06-4ca0-a5d6-ecb1853d2c56 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:25.509962 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-84934825-ec06-4ca0-a5d6-ecb1853d2c56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:25.510200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8bf71b4-7408-4421-bb6c-db769c4107f4 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:25.552046 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f39a4d9a-523f-4fdc-847b-d20963d49159 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:25.552687 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39a4d9a-523f-4fdc-847b-d20963d49159 None admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:25.552976 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39a4d9a-523f-4fdc-847b-d20963d49159 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:25.553259 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39a4d9a-523f-4fdc-847b-d20963d49159 None admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:25.553948 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f39a4d9a-523f-4fdc-847b-d20963d49159 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:25.555525 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f39a4d9a-523f-4fdc-847b-d20963d49159 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:25.594429 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 56/114] 10.222.0.25 () {60 vars in 1141 bytes} [Fri Oct 28 09:30:25 2022] POST /identity/v3/projects => generated 302 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:30:25.594429 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f39a4d9a-523f-4fdc-847b-d20963d49159 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:25.626438 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 59/115] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:30:25.626438 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84934825-ec06-4ca0-a5d6-ecb1853d2c56 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:25.691884 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:25.757195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84934825-ec06-4ca0-a5d6-ecb1853d2c56 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:25.807010 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f39a4d9a-523f-4fdc-847b-d20963d49159 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:25.811243 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4a8874-7c43-4aae-bec5-cf1b031c44dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:25.811902 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4a8874-7c43-4aae-bec5-cf1b031c44dc None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:25.812191 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4a8874-7c43-4aae-bec5-cf1b031c44dc None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:25.812472 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce4a8874-7c43-4aae-bec5-cf1b031c44dc None admin] PATH_INFO: `/v3/projects/6e28083c55ff4290af640f011fd2466f` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:25.813629 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce4a8874-7c43-4aae-bec5-cf1b031c44dc None admin] RBAC: Authorizing `identity:get_project(project_id=6e28083c55ff4290af640f011fd2466f)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:25.816285 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce4a8874-7c43-4aae-bec5-cf1b031c44dc None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:25.817247 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 60/116] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:30:25 2022] GET /identity/v3/projects/6e28083c55ff4290af640f011fd2466f => generated 298 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:25.817868 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4a8874-7c43-4aae-bec5-cf1b031c44dc None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:25.841289 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:25.864603 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c91e7122-c1a3-4f75-8d61-eaf56b8d5afd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:25.865259 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91e7122-c1a3-4f75-8d61-eaf56b8d5afd None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:25.865546 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91e7122-c1a3-4f75-8d61-eaf56b8d5afd None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:25.865828 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91e7122-c1a3-4f75-8d61-eaf56b8d5afd None admin] PATH_INFO: `/v3/roles/member` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:25.879643 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c91e7122-c1a3-4f75-8d61-eaf56b8d5afd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:30:25.885949 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-c91e7122-c1a3-4f75-8d61-eaf56b8d5afd None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 09:30:25.885949 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:30:25.885949 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:30:25.885949 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:30:25.885949 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:30:25.885949 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:30:25.885949 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:30:25.885949 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:30:25.885949 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:30:25.885949 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:30:25.885949 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:30:25.885949 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:30:25.885949 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:30:25.885949 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:30:25.885949 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:30:25.885949 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:30:25.886854 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:30:25.886854 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:30:25.886854 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:30:25.886854 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:30:25.886854 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:30:25.886854 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:30:25.886854 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:30:25.886854 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:30:25.886854 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:30:25.886854 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:30:25.886854 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:30:25.886854 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:30:25.886854 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:30:25.886854 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:30:25.886854 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:30:25.886854 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:30:25.887589 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:30:25.887589 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:30:25.887589 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:30:25.887589 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:30:25.887589 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:30:25.887589 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:30:25.887589 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:30:25.887589 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:30:25.887589 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:30:25.887589 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:30:25.887589 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 09:30:25.887589 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:30:25.887589 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 57/117] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:30:25 2022] GET /identity/v3/roles/member => generated 84 bytes in 301 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:30:25.888699 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308ece68-25f5-460e-9af1-64dea9f43663 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:25.888699 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308ece68-25f5-460e-9af1-64dea9f43663 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:25.888838 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308ece68-25f5-460e-9af1-64dea9f43663 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:25.890914 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 58/118] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:30:25.891655 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308ece68-25f5-460e-9af1-64dea9f43663 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:25.974574 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce4a8874-7c43-4aae-bec5-cf1b031c44dc None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:26.006760 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:26.009572 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-308ece68-25f5-460e-9af1-64dea9f43663 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:26.026673 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97334b0b-fc32-4998-b71c-17f981ecfc2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:26.027294 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97334b0b-fc32-4998-b71c-17f981ecfc2c None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:26.027578 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97334b0b-fc32-4998-b71c-17f981ecfc2c None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:26.027856 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97334b0b-fc32-4998-b71c-17f981ecfc2c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:26.053232 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b26a16-c838-49e5-b50c-c5b5bdeb9867 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:26.053919 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b26a16-c838-49e5-b50c-c5b5bdeb9867 None admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:26.054206 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b26a16-c838-49e5-b50c-c5b5bdeb9867 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:26.054519 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b26a16-c838-49e5-b50c-c5b5bdeb9867 None admin] PATH_INFO: `/v3/projects/6e28083c55ff4290af640f011fd2466f/users/8afbf6a033494d4cbbdea7a924ab5c63/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:26.057109 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88b26a16-c838-49e5-b50c-c5b5bdeb9867 None admin] RBAC: Authorizing `identity:create_grant(project_id=6e28083c55ff4290af640f011fd2466f, user_id=8afbf6a033494d4cbbdea7a924ab5c63, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:26.058604 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88b26a16-c838-49e5-b50c-c5b5bdeb9867 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:26.082133 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 59/119] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Oct 28 09:30:25 2022] PUT /identity/v3/projects/6e28083c55ff4290af640f011fd2466f/users/8afbf6a033494d4cbbdea7a924ab5c63/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:30:26.084186 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a0de00-5062-40b6-ab24-111b106c9533 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:26.084478 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a0de00-5062-40b6-ab24-111b106c9533 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:26.084811 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a0de00-5062-40b6-ab24-111b106c9533 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:26.105548 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-69a0de00-5062-40b6-ab24-111b106c9533 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:26.106897 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-69a0de00-5062-40b6-ab24-111b106c9533 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:26.127098 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97334b0b-fc32-4998-b71c-17f981ecfc2c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:26.129642 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97334b0b-fc32-4998-b71c-17f981ecfc2c None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:26.158046 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:26.203240 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 60/120] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:30:26.212132 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a0de00-5062-40b6-ab24-111b106c9533 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:26.285169 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 61/121] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:30:25 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 467 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 09:30:26.286387 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97334b0b-fc32-4998-b71c-17f981ecfc2c None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:26.311052 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:26.419713 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97334b0b-fc32-4998-b71c-17f981ecfc2c None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:26.428810 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a0de00-5062-40b6-ab24-111b106c9533 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:26.473724 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a58800-5736-4d12-8e0a-16a9c41990ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:26.475691 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a58800-5736-4d12-8e0a-16a9c41990ea None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:26.475691 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a58800-5736-4d12-8e0a-16a9c41990ea None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:26.475691 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a58800-5736-4d12-8e0a-16a9c41990ea None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:26.495994 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48a58800-5736-4d12-8e0a-16a9c41990ea None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:30:26.505534 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-48a58800-5736-4d12-8e0a-16a9c41990ea None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 09:30:26.505534 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:30:26.505534 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:30:26.505534 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:30:26.505534 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:30:26.505534 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:30:26.505534 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:30:26.505534 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:30:26.505534 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:30:26.505534 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:30:26.505534 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:30:26.505534 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:30:26.505534 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:30:26.505534 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:30:26.505534 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:30:26.505534 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:30:26.506493 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:30:26.506493 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:30:26.506493 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:30:26.506493 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:30:26.506493 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:30:26.506493 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:30:26.506493 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:30:26.506493 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:30:26.506493 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:30:26.506493 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:30:26.506493 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:30:26.506493 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:30:26.506493 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:30:26.506493 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:30:26.506493 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:30:26.506493 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:30:26.507256 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:30:26.507256 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:30:26.507256 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:30:26.507256 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:30:26.507256 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:30:26.507256 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:30:26.507256 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:30:26.507256 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:30:26.507256 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:30:26.507256 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:30:26.507256 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 09:30:26.507256 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:30:26.508868 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 62/122] 10.222.0.25 () {56 vars in 1114 bytes} [Fri Oct 28 09:30:26 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:30:26.510041 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a58800-5736-4d12-8e0a-16a9c41990ea None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:26.527927 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e01bcf5-c976-4189-9b90-4b11eba9b208 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:26.528570 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e01bcf5-c976-4189-9b90-4b11eba9b208 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:26.528856 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e01bcf5-c976-4189-9b90-4b11eba9b208 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:26.529136 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e01bcf5-c976-4189-9b90-4b11eba9b208 None admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:26.529828 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e01bcf5-c976-4189-9b90-4b11eba9b208 None admin] RBAC: Adding query filter params (name=member) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:26.530141 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e01bcf5-c976-4189-9b90-4b11eba9b208 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:26.532020 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e01bcf5-c976-4189-9b90-4b11eba9b208 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:26.542530 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 61/123] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Oct 28 09:30:26 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 339 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:26.543305 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e01bcf5-c976-4189-9b90-4b11eba9b208 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:26.596880 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:26.614985 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:26.644573 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a58800-5736-4d12-8e0a-16a9c41990ea None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:26.677990 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e01bcf5-c976-4189-9b90-4b11eba9b208 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:26.690649 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a742ca-9010-43a7-91c6-2a2660f8658c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:26.691598 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a742ca-9010-43a7-91c6-2a2660f8658c None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:26.691896 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a742ca-9010-43a7-91c6-2a2660f8658c None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:26.692183 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a742ca-9010-43a7-91c6-2a2660f8658c None admin] PATH_INFO: `/v3/projects/6e28083c55ff4290af640f011fd2466f` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:26.696339 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1a742ca-9010-43a7-91c6-2a2660f8658c None admin] RBAC: Authorizing `identity:get_project(project_id=6e28083c55ff4290af640f011fd2466f)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:26.697818 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1a742ca-9010-43a7-91c6-2a2660f8658c None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:26.700122 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 63/124] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:30:26 2022] GET /identity/v3/projects/6e28083c55ff4290af640f011fd2466f => generated 298 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:26.701822 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a742ca-9010-43a7-91c6-2a2660f8658c None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:26.730419 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fec28e0-b7ff-4e43-8124-4e43ddf94e91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:26.730419 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fec28e0-b7ff-4e43-8124-4e43ddf94e91 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:26.730419 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fec28e0-b7ff-4e43-8124-4e43ddf94e91 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:26.730419 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fec28e0-b7ff-4e43-8124-4e43ddf94e91 None admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:26.730806 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fec28e0-b7ff-4e43-8124-4e43ddf94e91 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:26.730806 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fec28e0-b7ff-4e43-8124-4e43ddf94e91 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:26.731848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fec28e0-b7ff-4e43-8124-4e43ddf94e91 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:26.742725 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 62/125] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:30:26 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:26.743653 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fec28e0-b7ff-4e43-8124-4e43ddf94e91 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:26.852393 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a742ca-9010-43a7-91c6-2a2660f8658c None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:26.886234 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:26.900297 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fec28e0-b7ff-4e43-8124-4e43ddf94e91 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:26.908016 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648ba4ce-e32a-42fb-ae44-e38adb6d23fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:26.909549 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648ba4ce-e32a-42fb-ae44-e38adb6d23fd None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:26.910189 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648ba4ce-e32a-42fb-ae44-e38adb6d23fd None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:26.910697 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-648ba4ce-e32a-42fb-ae44-e38adb6d23fd None admin] PATH_INFO: `/v3/roles/member` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:26.925249 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-648ba4ce-e32a-42fb-ae44-e38adb6d23fd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:30:26.931536 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-648ba4ce-e32a-42fb-ae44-e38adb6d23fd None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 09:30:26.931536 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:30:26.931536 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:30:26.931536 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:30:26.931536 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:30:26.931536 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:30:26.931536 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:30:26.931536 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:30:26.931536 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:30:26.931536 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:30:26.931536 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:30:26.931536 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:30:26.931536 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:30:26.931536 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:30:26.931536 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:30:26.931536 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:30:26.932414 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:30:26.932414 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:30:26.932414 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:30:26.932414 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:30:26.932414 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:30:26.932414 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:30:26.932414 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:30:26.932414 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:30:26.932414 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:30:26.932414 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:30:26.932414 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:30:26.932414 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:30:26.932414 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:30:26.932414 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:30:26.932414 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:30:26.932414 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:30:26.933162 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:30:26.933162 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:30:26.933162 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:30:26.933162 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:30:26.933162 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:30:26.933162 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:30:26.933162 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:30:26.933162 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:30:26.933162 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:30:26.933162 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:30:26.933162 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 09:30:26.933162 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:30:26.935852 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 64/126] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:30:26 2022] GET /identity/v3/roles/member => generated 84 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:30:26.937329 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648ba4ce-e32a-42fb-ae44-e38adb6d23fd None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:26.947243 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152f2f6d-4054-403e-849d-6b611c15caf0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:26.948200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152f2f6d-4054-403e-849d-6b611c15caf0 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:26.948594 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152f2f6d-4054-403e-849d-6b611c15caf0 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:26.948994 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152f2f6d-4054-403e-849d-6b611c15caf0 None admin] PATH_INFO: `/v3/users/c717178de72f493788554ab410b527de` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:26.950252 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:26.950367 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-152f2f6d-4054-403e-849d-6b611c15caf0 None admin] RBAC: Authorizing `identity:get_user(user_id=c717178de72f493788554ab410b527de)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:26.951971 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-152f2f6d-4054-403e-849d-6b611c15caf0 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:26.958442 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 63/127] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:30:26 2022] GET /identity/v3/users/c717178de72f493788554ab410b527de => generated 275 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:26.958442 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152f2f6d-4054-403e-849d-6b611c15caf0 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:27.076739 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-648ba4ce-e32a-42fb-ae44-e38adb6d23fd None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:27.099631 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152f2f6d-4054-403e-849d-6b611c15caf0 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:27.128427 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e414277-1f1a-4790-a2f1-bfa20b935a47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:27.129529 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e414277-1f1a-4790-a2f1-bfa20b935a47 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:27.130031 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e414277-1f1a-4790-a2f1-bfa20b935a47 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:27.130544 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e414277-1f1a-4790-a2f1-bfa20b935a47 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:27.132151 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e414277-1f1a-4790-a2f1-bfa20b935a47 None admin] RBAC: Adding query filter params (role.id=12f06a0e43d345aba46a51c26c012ce3, scope.project.id=6e28083c55ff4290af640f011fd2466f, user.id=d287c1589ecb4e1caa5ee5f36033e00e) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:27.132998 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e414277-1f1a-4790-a2f1-bfa20b935a47 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:27.135330 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e414277-1f1a-4790-a2f1-bfa20b935a47 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:27.145897 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 65/128] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:30:26 2022] GET /identity/v3/role_assignments?user.id=d287c1589ecb4e1caa5ee5f36033e00e&scope.project.id=6e28083c55ff4290af640f011fd2466f&role.id=12f06a0e43d345aba46a51c26c012ce3 => generated 259 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:27.146157 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f6857c5-dfe4-4234-93d6-66e5dd810cea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:27.146846 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6857c5-dfe4-4234-93d6-66e5dd810cea None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:27.147138 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6857c5-dfe4-4234-93d6-66e5dd810cea None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:27.147426 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6857c5-dfe4-4234-93d6-66e5dd810cea None admin] PATH_INFO: `/v3/users/c717178de72f493788554ab410b527de` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:27.147961 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e414277-1f1a-4790-a2f1-bfa20b935a47 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:27.155906 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f6857c5-dfe4-4234-93d6-66e5dd810cea None admin] RBAC: Authorizing `identity:get_user(user_id=c717178de72f493788554ab410b527de)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:27.157716 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f6857c5-dfe4-4234-93d6-66e5dd810cea None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:27.159807 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 64/129] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:30:26 2022] GET /identity/v3/users/c717178de72f493788554ab410b527de => generated 275 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:27.160514 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f6857c5-dfe4-4234-93d6-66e5dd810cea None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:27.216260 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:27.240261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:27.280031 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e414277-1f1a-4790-a2f1-bfa20b935a47 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:27.322100 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30be5a86-355a-4df1-beaf-4564608f62b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:27.323105 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30be5a86-355a-4df1-beaf-4564608f62b3 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:27.323687 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30be5a86-355a-4df1-beaf-4564608f62b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:27.324160 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30be5a86-355a-4df1-beaf-4564608f62b3 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:27.325288 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30be5a86-355a-4df1-beaf-4564608f62b3 None admin] RBAC: Adding query filter params (name=member) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:27.325796 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30be5a86-355a-4df1-beaf-4564608f62b3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:27.327466 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30be5a86-355a-4df1-beaf-4564608f62b3 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:27.337698 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 66/130] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Oct 28 09:30:27 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:27.339323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30be5a86-355a-4df1-beaf-4564608f62b3 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:27.399094 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f6857c5-dfe4-4234-93d6-66e5dd810cea None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:27.458729 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33bc6211-23c1-486c-a4ff-57ad286614bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:27.459380 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33bc6211-23c1-486c-a4ff-57ad286614bb None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:27.459666 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33bc6211-23c1-486c-a4ff-57ad286614bb None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:27.459942 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33bc6211-23c1-486c-a4ff-57ad286614bb None admin] PATH_INFO: `/v3/projects/6e28083c55ff4290af640f011fd2466f` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:27.461106 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33bc6211-23c1-486c-a4ff-57ad286614bb None admin] RBAC: Authorizing `identity:get_project(project_id=6e28083c55ff4290af640f011fd2466f)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:27.462676 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33bc6211-23c1-486c-a4ff-57ad286614bb None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:27.464343 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 65/131] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:30:27 2022] GET /identity/v3/projects/6e28083c55ff4290af640f011fd2466f => generated 298 bytes in 304 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:27.465129 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33bc6211-23c1-486c-a4ff-57ad286614bb None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:27.564026 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30be5a86-355a-4df1-beaf-4564608f62b3 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:27.578430 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33bc6211-23c1-486c-a4ff-57ad286614bb None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:27.582343 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:27.615657 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89509944-c916-420e-9173-c36f26fc7b2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:27.616368 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89509944-c916-420e-9173-c36f26fc7b2d None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:27.616649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89509944-c916-420e-9173-c36f26fc7b2d None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:27.616926 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89509944-c916-420e-9173-c36f26fc7b2d None admin] PATH_INFO: `/v3/users/c717178de72f493788554ab410b527de` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:27.622419 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89509944-c916-420e-9173-c36f26fc7b2d None admin] RBAC: Authorizing `identity:get_user(user_id=c717178de72f493788554ab410b527de)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:27.622419 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89509944-c916-420e-9173-c36f26fc7b2d None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:27.622419 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 66/132] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:30:27 2022] GET /identity/v3/users/c717178de72f493788554ab410b527de => generated 275 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:27.626770 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89509944-c916-420e-9173-c36f26fc7b2d None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:27.683886 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:27.711128 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33dd347f-30e3-4629-8d22-5053b99ab562 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:27.712330 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33dd347f-30e3-4629-8d22-5053b99ab562 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:27.712616 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33dd347f-30e3-4629-8d22-5053b99ab562 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:27.712895 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33dd347f-30e3-4629-8d22-5053b99ab562 None admin] PATH_INFO: `/v3/projects/6e28083c55ff4290af640f011fd2466f` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:27.719139 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33dd347f-30e3-4629-8d22-5053b99ab562 None admin] RBAC: Authorizing `identity:get_project(project_id=6e28083c55ff4290af640f011fd2466f)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:27.720623 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33dd347f-30e3-4629-8d22-5053b99ab562 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:27.722307 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 67/133] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:30:27 2022] GET /identity/v3/projects/6e28083c55ff4290af640f011fd2466f => generated 298 bytes in 384 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:27.731289 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33dd347f-30e3-4629-8d22-5053b99ab562 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:27.732219 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89509944-c916-420e-9173-c36f26fc7b2d None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:27.824062 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da8c8a1-a6ea-4865-bbfd-090df893e0c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:27.824764 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da8c8a1-a6ea-4865-bbfd-090df893e0c4 None admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:27.825539 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da8c8a1-a6ea-4865-bbfd-090df893e0c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:27.825539 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da8c8a1-a6ea-4865-bbfd-090df893e0c4 None admin] PATH_INFO: `/v3/projects/6e28083c55ff4290af640f011fd2466f/users/c717178de72f493788554ab410b527de/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:27.828904 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7da8c8a1-a6ea-4865-bbfd-090df893e0c4 None admin] RBAC: Authorizing `identity:create_grant(project_id=6e28083c55ff4290af640f011fd2466f, user_id=c717178de72f493788554ab410b527de, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:27.834421 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7da8c8a1-a6ea-4865-bbfd-090df893e0c4 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:27.860346 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 67/134] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Oct 28 09:30:27 2022] PUT /identity/v3/projects/6e28083c55ff4290af640f011fd2466f/users/c717178de72f493788554ab410b527de/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 238 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:30:27.861889 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da8c8a1-a6ea-4865-bbfd-090df893e0c4 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:27.921011 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:27.968733 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33dd347f-30e3-4629-8d22-5053b99ab562 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:28.076745 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873fea69-9b41-48b8-8adb-cdf36a452fbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:28.077622 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873fea69-9b41-48b8-8adb-cdf36a452fbe None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:28.078010 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873fea69-9b41-48b8-8adb-cdf36a452fbe None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:28.078416 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-873fea69-9b41-48b8-8adb-cdf36a452fbe None admin] PATH_INFO: `/v3/projects/f075fea87a3c4191a9ad2ed506c8f6a3` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:28.083432 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-873fea69-9b41-48b8-8adb-cdf36a452fbe None admin] RBAC: Authorizing `identity:get_project(project_id=f075fea87a3c4191a9ad2ed506c8f6a3)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:28.085140 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-873fea69-9b41-48b8-8adb-cdf36a452fbe None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:28.087358 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 68/135] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:30:27 2022] GET /identity/v3/projects/f075fea87a3c4191a9ad2ed506c8f6a3 => generated 312 bytes in 364 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:28.095425 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da8c8a1-a6ea-4865-bbfd-090df893e0c4 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:28.106423 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873fea69-9b41-48b8-8adb-cdf36a452fbe None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:28.143415 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67475cd7-b522-4db6-bb3d-c7370c13b954 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:28.144113 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67475cd7-b522-4db6-bb3d-c7370c13b954 None admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:28.144400 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67475cd7-b522-4db6-bb3d-c7370c13b954 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:28.144681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67475cd7-b522-4db6-bb3d-c7370c13b954 None admin] PATH_INFO: `/v3/projects/6e28083c55ff4290af640f011fd2466f/users/c717178de72f493788554ab410b527de/roles/d93b4c94b590457199c3d301f947b0fb` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:28.147552 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67475cd7-b522-4db6-bb3d-c7370c13b954 None admin] RBAC: Authorizing `identity:create_grant(project_id=6e28083c55ff4290af640f011fd2466f, user_id=c717178de72f493788554ab410b527de, role_id=d93b4c94b590457199c3d301f947b0fb)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:28.149431 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67475cd7-b522-4db6-bb3d-c7370c13b954 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:28.180473 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 68/136] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Oct 28 09:30:27 2022] PUT /identity/v3/projects/6e28083c55ff4290af640f011fd2466f/users/c717178de72f493788554ab410b527de/roles/d93b4c94b590457199c3d301f947b0fb => generated 0 bytes in 320 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:30:28.254969 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:28.285694 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3011bb4-be25-4161-b5ae-6be4937f5305 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:28.286246 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3011bb4-be25-4161-b5ae-6be4937f5305 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:28.286704 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3011bb4-be25-4161-b5ae-6be4937f5305 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:28.288518 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 69/137] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:28 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 09:30:28.298449 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6acba27d-66b6-4cf7-b9a5-801d3174ca54 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:28.299007 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6acba27d-66b6-4cf7-b9a5-801d3174ca54 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:28.299442 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6acba27d-66b6-4cf7-b9a5-801d3174ca54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:28.327508 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-6acba27d-66b6-4cf7-b9a5-801d3174ca54 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:28.329966 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-6acba27d-66b6-4cf7-b9a5-801d3174ca54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:28.347331 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-873fea69-9b41-48b8-8adb-cdf36a452fbe None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:28.440433 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-663a19cc-d8a3-4d11-b7a0-c31c87d499cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:28.441392 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663a19cc-d8a3-4d11-b7a0-c31c87d499cb None admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:28.441815 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663a19cc-d8a3-4d11-b7a0-c31c87d499cb None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:28.442227 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663a19cc-d8a3-4d11-b7a0-c31c87d499cb None admin] PATH_INFO: `/v3/projects/f075fea87a3c4191a9ad2ed506c8f6a3/users/c717178de72f493788554ab410b527de/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:28.448935 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-663a19cc-d8a3-4d11-b7a0-c31c87d499cb None admin] RBAC: Authorizing `identity:create_grant(project_id=f075fea87a3c4191a9ad2ed506c8f6a3, user_id=c717178de72f493788554ab410b527de, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:28.450737 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-663a19cc-d8a3-4d11-b7a0-c31c87d499cb None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:28.481462 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 69/138] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Oct 28 09:30:28 2022] PUT /identity/v3/projects/f075fea87a3c4191a9ad2ed506c8f6a3/users/c717178de72f493788554ab410b527de/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 376 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:30:28.525553 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 70/139] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 09:30:28.705361 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e2b6db-1ee3-4fe1-bb75-103f92985aea None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:28.705863 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e2b6db-1ee3-4fe1-bb75-103f92985aea None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:28.706350 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e2b6db-1ee3-4fe1-bb75-103f92985aea None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:28.708812 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 70/140] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:28 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:30:28.719004 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08062731-e104-4ac3-a89f-17c5661704ba None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:28.719498 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08062731-e104-4ac3-a89f-17c5661704ba None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:28.719976 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08062731-e104-4ac3-a89f-17c5661704ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:28.763432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-08062731-e104-4ac3-a89f-17c5661704ba None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:28.764639 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-08062731-e104-4ac3-a89f-17c5661704ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:28.782212 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:28.862342 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 71/141] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:30:29.107958 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e2b6db-1ee3-4fe1-bb75-103f92985aea None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:29.311978 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e2b6db-1ee3-4fe1-bb75-103f92985aea None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:29.328992 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08062731-e104-4ac3-a89f-17c5661704ba None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:29.366843 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:29.381796 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20658f66-428d-4852-87ec-0befeecb6f7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:29.382804 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20658f66-428d-4852-87ec-0befeecb6f7f None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:29.383351 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20658f66-428d-4852-87ec-0befeecb6f7f None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:29.383844 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20658f66-428d-4852-87ec-0befeecb6f7f None admin] PATH_INFO: `/v3/domains/default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:29.386544 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20658f66-428d-4852-87ec-0befeecb6f7f None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:29.388423 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20658f66-428d-4852-87ec-0befeecb6f7f None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:29.390726 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 71/142] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:30:29 2022] GET /identity/v3/domains/default => generated 199 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:29.398464 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20658f66-428d-4852-87ec-0befeecb6f7f None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:29.618657 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08062731-e104-4ac3-a89f-17c5661704ba None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:29.635246 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20658f66-428d-4852-87ec-0befeecb6f7f None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:29.707170 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c8f087-fb73-43d3-9657-f8e57defeada None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:29.707831 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c8f087-fb73-43d3-9657-f8e57defeada None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:29.708877 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c8f087-fb73-43d3-9657-f8e57defeada None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:29.708877 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c8f087-fb73-43d3-9657-f8e57defeada None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:29.737518 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf86d9de-2008-460f-a8d4-f0790171f892 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:29.738162 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf86d9de-2008-460f-a8d4-f0790171f892 None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:29.738469 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf86d9de-2008-460f-a8d4-f0790171f892 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:29.738755 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf86d9de-2008-460f-a8d4-f0790171f892 None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:29.739435 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf86d9de-2008-460f-a8d4-f0790171f892 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:29.741248 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf86d9de-2008-460f-a8d4-f0790171f892 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:29.744064 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24c8f087-fb73-43d3-9657-f8e57defeada None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:30:29.771628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-24c8f087-fb73-43d3-9657-f8e57defeada None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:30:29.771628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:30:29.771628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:30:29.771628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:30:29.771628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:30:29.771628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:30:29.771628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:30:29.771628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:30:29.771628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:30:29.771628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:30:29.771628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:30:29.771628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:30:29.771628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:30:29.771628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:30:29.771628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:30:29.771628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:30:29.772535 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:30:29.772535 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:30:29.772535 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:30:29.772535 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:30:29.772535 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:30:29.772535 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:30:29.772535 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:30:29.772535 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:30:29.772535 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:30:29.772535 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:30:29.772535 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:30:29.772535 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:30:29.772535 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:30:29.772535 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:30:29.772535 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:30:29.772535 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:30:29.773376 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:30:29.773376 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:30:29.773376 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:30:29.773376 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:30:29.773376 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:30:29.773376 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:30:29.773376 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:30:29.773376 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:30:29.773376 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:30:29.773376 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:30:29.773376 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:30:29.773376 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:30:29.773376 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 72/143] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Oct 28 09:30:29 2022] GET /identity/v3/roles/admin => generated 83 bytes in 444 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:30:29.782829 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c8f087-fb73-43d3-9657-f8e57defeada None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:29.793062 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 72/144] 10.222.0.25 () {60 vars in 1132 bytes} [Fri Oct 28 09:30:29 2022] POST /identity/v3/users => generated 283 bytes in 395 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:30:29.800944 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:30.110597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c8f087-fb73-43d3-9657-f8e57defeada None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:30.189683 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c631e1d3-3598-439a-8c9c-84a00fd6f5ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:30.190322 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c631e1d3-3598-439a-8c9c-84a00fd6f5ba None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:30.190624 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c631e1d3-3598-439a-8c9c-84a00fd6f5ba None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:30.190901 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c631e1d3-3598-439a-8c9c-84a00fd6f5ba None admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:30.191583 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c631e1d3-3598-439a-8c9c-84a00fd6f5ba None admin] RBAC: Adding query filter params (name=admin) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:30.191885 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c631e1d3-3598-439a-8c9c-84a00fd6f5ba None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:30.193282 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c631e1d3-3598-439a-8c9c-84a00fd6f5ba None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:30.214006 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 73/145] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:30:29 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 431 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:30:30.219772 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf86d9de-2008-460f-a8d4-f0790171f892 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:30.313084 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:30.373701 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf86d9de-2008-460f-a8d4-f0790171f892 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:30.427997 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec90f134-53b4-4587-bb63-a42c8a6e0da5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:30.428664 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec90f134-53b4-4587-bb63-a42c8a6e0da5 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:30.428951 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec90f134-53b4-4587-bb63-a42c8a6e0da5 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:30.429230 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec90f134-53b4-4587-bb63-a42c8a6e0da5 None admin] PATH_INFO: `/v3/users/8afbf6a033494d4cbbdea7a924ab5c63` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:30.435176 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec90f134-53b4-4587-bb63-a42c8a6e0da5 None admin] RBAC: Authorizing `identity:get_user(user_id=8afbf6a033494d4cbbdea7a924ab5c63)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:30.436660 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec90f134-53b4-4587-bb63-a42c8a6e0da5 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:30.438367 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 73/146] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:30:30 2022] GET /identity/v3/users/8afbf6a033494d4cbbdea7a924ab5c63 => generated 247 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:30:30.443063 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b0cdd1-265e-4e41-98f0-f3e3075c69ab None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:30.443549 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b0cdd1-265e-4e41-98f0-f3e3075c69ab None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:30.443635 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5b0cdd1-265e-4e41-98f0-f3e3075c69ab None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:30.445011 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 74/147] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:30:30.446514 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec90f134-53b4-4587-bb63-a42c8a6e0da5 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:30.453646 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e9325e-b6a1-40bf-8472-d69e5deb7a1b None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:30.453929 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e9325e-b6a1-40bf-8472-d69e5deb7a1b None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:30.454199 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e9325e-b6a1-40bf-8472-d69e5deb7a1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:30.491624 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-73e9325e-b6a1-40bf-8472-d69e5deb7a1b None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:30.492929 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-73e9325e-b6a1-40bf-8472-d69e5deb7a1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:30.638858 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:30.646563 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 75/148] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 09:30:30.665421 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec90f134-53b4-4587-bb63-a42c8a6e0da5 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:30.695858 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:30.712617 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ef6868-74e0-4a81-9704-bb2410e3347f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:30.713254 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ef6868-74e0-4a81-9704-bb2410e3347f None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:30.713544 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ef6868-74e0-4a81-9704-bb2410e3347f None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:30.713839 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ef6868-74e0-4a81-9704-bb2410e3347f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:30.874112 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79ef6868-74e0-4a81-9704-bb2410e3347f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:30.875653 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79ef6868-74e0-4a81-9704-bb2410e3347f None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:31.052500 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 74/149] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:30:30 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 607 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 09:30:31.065772 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e9325e-b6a1-40bf-8472-d69e5deb7a1b None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:31.134540 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ef6868-74e0-4a81-9704-bb2410e3347f None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:31.182877 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:31.319032 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e9325e-b6a1-40bf-8472-d69e5deb7a1b None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:31.345809 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ef6868-74e0-4a81-9704-bb2410e3347f None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:31.400892 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49b2371-4474-40ce-a26a-52360f0e63ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:31.401799 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49b2371-4474-40ce-a26a-52360f0e63ab None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:31.402243 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49b2371-4474-40ce-a26a-52360f0e63ab None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:31.402714 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49b2371-4474-40ce-a26a-52360f0e63ab None admin] PATH_INFO: `/v3/projects/6e28083c55ff4290af640f011fd2466f` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:31.404714 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f49b2371-4474-40ce-a26a-52360f0e63ab None admin] RBAC: Authorizing `identity:get_project(project_id=6e28083c55ff4290af640f011fd2466f)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:31.406407 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f49b2371-4474-40ce-a26a-52360f0e63ab None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:31.408654 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 76/150] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:30:31 2022] GET /identity/v3/projects/6e28083c55ff4290af640f011fd2466f => generated 298 bytes in 343 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:31.412871 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ce602a-2fcc-4d54-8bd9-288ebda05638 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:31.414224 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ce602a-2fcc-4d54-8bd9-288ebda05638 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:31.415741 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ce602a-2fcc-4d54-8bd9-288ebda05638 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:31.416613 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9fb8ab7-e46a-43e2-880f-9e5da3eaa67c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:31.417503 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9fb8ab7-e46a-43e2-880f-9e5da3eaa67c None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:31.417944 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9fb8ab7-e46a-43e2-880f-9e5da3eaa67c None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:31.418365 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9fb8ab7-e46a-43e2-880f-9e5da3eaa67c None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:31.420719 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 77/151] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:31 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:30:31.423031 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ce602a-2fcc-4d54-8bd9-288ebda05638 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:31.449558 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9fb8ab7-e46a-43e2-880f-9e5da3eaa67c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:30:31.454821 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-e9fb8ab7-e46a-43e2-880f-9e5da3eaa67c None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 09:30:31.454821 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:30:31.454821 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:30:31.454821 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:30:31.454821 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:30:31.454821 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:30:31.454821 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:30:31.454821 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:30:31.454821 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:30:31.454821 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:30:31.454821 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:30:31.454821 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:30:31.454821 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:30:31.454821 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:30:31.454821 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:30:31.454821 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:30:31.455719 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:30:31.455719 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:30:31.455719 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:30:31.455719 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:30:31.455719 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:30:31.455719 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:30:31.455719 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:30:31.455719 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:30:31.455719 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:30:31.455719 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:30:31.455719 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:30:31.455719 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:30:31.455719 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:30:31.455719 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:30:31.455719 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:30:31.455719 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:30:31.456591 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:30:31.456591 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:30:31.456591 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:30:31.456591 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:30:31.456591 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:30:31.456591 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:30:31.456591 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:30:31.456591 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:30:31.456591 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:30:31.456591 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:30:31.456591 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 09:30:31.456591 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:30:31.458451 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 75/152] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:30:31 2022] GET /identity/v3/roles/reader => generated 84 bytes in 325 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:30:31.464671 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71153d54-57f5-4720-a0e5-994260ccd3fe None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:31.465167 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71153d54-57f5-4720-a0e5-994260ccd3fe None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:31.465585 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71153d54-57f5-4720-a0e5-994260ccd3fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:31.514352 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-71153d54-57f5-4720-a0e5-994260ccd3fe None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:31.522203 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-71153d54-57f5-4720-a0e5-994260ccd3fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:31.584066 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:31.682040 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ce602a-2fcc-4d54-8bd9-288ebda05638 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:31.706748 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 76/153] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 243 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:30:31.707992 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71153d54-57f5-4720-a0e5-994260ccd3fe None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:31.740363 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:31.754373 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf261cd-d133-40c2-9fed-67d76c3a7ea8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:31.755038 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf261cd-d133-40c2-9fed-67d76c3a7ea8 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:31.755382 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf261cd-d133-40c2-9fed-67d76c3a7ea8 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:31.755657 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf261cd-d133-40c2-9fed-67d76c3a7ea8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:31.757841 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf261cd-d133-40c2-9fed-67d76c3a7ea8 None admin] RBAC: Adding query filter params (role.id=e13d798a98ea4c3dbb87a236ec683d32, scope.project.id=6e28083c55ff4290af640f011fd2466f, user.id=8afbf6a033494d4cbbdea7a924ab5c63) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:31.758265 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf261cd-d133-40c2-9fed-67d76c3a7ea8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:31.762462 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf261cd-d133-40c2-9fed-67d76c3a7ea8 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:31.785792 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 78/154] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:30:31 2022] GET /identity/v3/role_assignments?user.id=8afbf6a033494d4cbbdea7a924ab5c63&scope.project.id=6e28083c55ff4290af640f011fd2466f&role.id=e13d798a98ea4c3dbb87a236ec683d32 => generated 608 bytes in 363 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:31.949062 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71153d54-57f5-4720-a0e5-994260ccd3fe None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:32.016614 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3325e689-7064-4cab-974b-4df092304331 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:32.016819 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3325e689-7064-4cab-974b-4df092304331 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:32.017090 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3325e689-7064-4cab-974b-4df092304331 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:32.023013 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 79/155] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:32 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 09:30:32.025362 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b7cb0e-87d1-426d-b7b2-825be8f35f2d None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:32.025579 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b7cb0e-87d1-426d-b7b2-825be8f35f2d None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:32.025848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b7cb0e-87d1-426d-b7b2-825be8f35f2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:32.059154 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972f2621-e7f1-4890-ad20-4b8678409fad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:32.059912 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972f2621-e7f1-4890-ad20-4b8678409fad None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:32.060208 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972f2621-e7f1-4890-ad20-4b8678409fad None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:32.060490 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972f2621-e7f1-4890-ad20-4b8678409fad None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:32.061185 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972f2621-e7f1-4890-ad20-4b8678409fad None admin] RBAC: Adding query filter params (name=reader) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:32.061495 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972f2621-e7f1-4890-ad20-4b8678409fad None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:32.062932 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972f2621-e7f1-4890-ad20-4b8678409fad None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:32.063453 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-22b7cb0e-87d1-426d-b7b2-825be8f35f2d None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:32.066502 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-22b7cb0e-87d1-426d-b7b2-825be8f35f2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:32.078238 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 77/156] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Oct 28 09:30:31 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 370 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:32.083810 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972f2621-e7f1-4890-ad20-4b8678409fad None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:32.213242 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 80/157] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:30:32.214850 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b7cb0e-87d1-426d-b7b2-825be8f35f2d None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:32.343437 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:32.389672 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:32.400579 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972f2621-e7f1-4890-ad20-4b8678409fad None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:32.454536 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b7cb0e-87d1-426d-b7b2-825be8f35f2d None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:32.484997 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe405e87-4d89-4d98-b45d-58d2d52b83cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:32.485918 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe405e87-4d89-4d98-b45d-58d2d52b83cd None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:32.487782 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe405e87-4d89-4d98-b45d-58d2d52b83cd None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:32.489740 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe405e87-4d89-4d98-b45d-58d2d52b83cd None admin] PATH_INFO: `/v3/users/d287c1589ecb4e1caa5ee5f36033e00e` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:32.491216 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe405e87-4d89-4d98-b45d-58d2d52b83cd None admin] RBAC: Authorizing `identity:get_user(user_id=d287c1589ecb4e1caa5ee5f36033e00e)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:32.495072 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe405e87-4d89-4d98-b45d-58d2d52b83cd None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:32.498027 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 78/158] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:30:32 2022] GET /identity/v3/users/d287c1589ecb4e1caa5ee5f36033e00e => generated 289 bytes in 415 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:30:32.505815 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ca2eb1-a586-4c62-b11d-b3f96dc6f5da None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:32.511094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ca2eb1-a586-4c62-b11d-b3f96dc6f5da None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:32.511603 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ca2eb1-a586-4c62-b11d-b3f96dc6f5da None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:32.513490 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 79/159] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:32 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:30:32.524751 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be0acaa-fa2e-4a80-8244-a110ad401c4a None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:32.525270 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be0acaa-fa2e-4a80-8244-a110ad401c4a None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:32.525713 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be0acaa-fa2e-4a80-8244-a110ad401c4a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:32.537624 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594baa1c-1c4f-4a93-9ca5-0ef1c79efdb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:32.553971 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594baa1c-1c4f-4a93-9ca5-0ef1c79efdb1 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:32.554614 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594baa1c-1c4f-4a93-9ca5-0ef1c79efdb1 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:32.555089 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594baa1c-1c4f-4a93-9ca5-0ef1c79efdb1 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:32.569370 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-6be0acaa-fa2e-4a80-8244-a110ad401c4a None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:32.572655 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-6be0acaa-fa2e-4a80-8244-a110ad401c4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:32.578628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-594baa1c-1c4f-4a93-9ca5-0ef1c79efdb1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:30:32.597264 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-594baa1c-1c4f-4a93-9ca5-0ef1c79efdb1 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 09:30:32.597264 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:30:32.597264 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:30:32.597264 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:30:32.597264 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:30:32.597264 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:30:32.597264 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:30:32.597264 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:30:32.597264 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:30:32.597264 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:30:32.597264 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:30:32.597264 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:30:32.597264 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:30:32.597264 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:30:32.597264 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:30:32.597264 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:30:32.598654 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:30:32.598654 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:30:32.598654 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:30:32.598654 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:30:32.598654 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:30:32.598654 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:30:32.598654 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:30:32.598654 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:30:32.598654 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:30:32.598654 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:30:32.598654 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:30:32.598654 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:30:32.598654 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:30:32.598654 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:30:32.598654 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:30:32.598654 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:30:32.599449 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:30:32.599449 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:30:32.599449 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:30:32.599449 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:30:32.599449 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:30:32.599449 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:30:32.599449 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:30:32.599449 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:30:32.599449 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:30:32.599449 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:30:32.599449 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 09:30:32.599449 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:30:32.605046 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 81/160] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:30:32 2022] GET /identity/v3/roles/member => generated 84 bytes in 391 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:30:32.606308 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594baa1c-1c4f-4a93-9ca5-0ef1c79efdb1 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:32.762126 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 80/161] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 239 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 09:30:32.763394 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be0acaa-fa2e-4a80-8244-a110ad401c4a None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:32.931000 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594baa1c-1c4f-4a93-9ca5-0ef1c79efdb1 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:32.981081 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:32.998811 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:33.015739 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd3d7a95-72ed-4bc0-9c7c-1f862a96c553 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:33.019121 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3d7a95-72ed-4bc0-9c7c-1f862a96c553 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:33.019518 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3d7a95-72ed-4bc0-9c7c-1f862a96c553 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:33.019804 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3d7a95-72ed-4bc0-9c7c-1f862a96c553 None admin] PATH_INFO: `/v3/projects/6e28083c55ff4290af640f011fd2466f` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:33.022863 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd3d7a95-72ed-4bc0-9c7c-1f862a96c553 None admin] RBAC: Authorizing `identity:get_project(project_id=6e28083c55ff4290af640f011fd2466f)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:33.024885 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd3d7a95-72ed-4bc0-9c7c-1f862a96c553 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:33.036152 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 82/162] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:30:32 2022] GET /identity/v3/projects/6e28083c55ff4290af640f011fd2466f => generated 298 bytes in 430 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:33.037119 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd3d7a95-72ed-4bc0-9c7c-1f862a96c553 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:33.092567 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be0acaa-fa2e-4a80-8244-a110ad401c4a None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:33.177629 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd3d7a95-72ed-4bc0-9c7c-1f862a96c553 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:33.215828 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080f87e8-0628-4d79-98a2-e7793a49ed79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:33.219328 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080f87e8-0628-4d79-98a2-e7793a49ed79 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:33.219328 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080f87e8-0628-4d79-98a2-e7793a49ed79 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:33.219328 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-080f87e8-0628-4d79-98a2-e7793a49ed79 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:33.219328 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-080f87e8-0628-4d79-98a2-e7793a49ed79 None admin] RBAC: Adding query filter params (name=member) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:33.219328 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-080f87e8-0628-4d79-98a2-e7793a49ed79 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:33.220327 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-080f87e8-0628-4d79-98a2-e7793a49ed79 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:33.238543 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3872a1be-bd26-42ca-a9db-f0cf2bd78392 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:33.239414 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3872a1be-bd26-42ca-a9db-f0cf2bd78392 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:33.239707 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3872a1be-bd26-42ca-a9db-f0cf2bd78392 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:33.240076 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3872a1be-bd26-42ca-a9db-f0cf2bd78392 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:33.240939 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 81/163] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Oct 28 09:30:32 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 478 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:33.241941 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080f87e8-0628-4d79-98a2-e7793a49ed79 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:33.261386 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3872a1be-bd26-42ca-a9db-f0cf2bd78392 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:30:33.271919 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-3872a1be-bd26-42ca-a9db-f0cf2bd78392 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 09:30:33.271919 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:30:33.271919 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:30:33.271919 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:30:33.271919 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:30:33.271919 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:30:33.271919 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:30:33.271919 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:30:33.271919 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:30:33.271919 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:30:33.271919 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:30:33.271919 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:30:33.271919 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:30:33.271919 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:30:33.271919 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:30:33.271919 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:30:33.272803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:30:33.272803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:30:33.272803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:30:33.272803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:30:33.272803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:30:33.272803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:30:33.272803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:30:33.272803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:30:33.272803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:30:33.272803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:30:33.272803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:30:33.272803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:30:33.272803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:30:33.272803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:30:33.272803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:30:33.272803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:30:33.273575 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:30:33.273575 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:30:33.273575 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:30:33.273575 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:30:33.273575 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:30:33.273575 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:30:33.273575 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:30:33.273575 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:30:33.273575 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:30:33.273575 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:30:33.273575 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 09:30:33.273575 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:30:33.275491 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 83/164] 10.222.0.25 () {56 vars in 1114 bytes} [Fri Oct 28 09:30:33 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:30:33.279253 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3872a1be-bd26-42ca-a9db-f0cf2bd78392 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:33.371111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:33.427997 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3872a1be-bd26-42ca-a9db-f0cf2bd78392 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:33.506356 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f82978c-d4bf-4d7b-b609-cc5fd4d55c89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:33.507067 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f82978c-d4bf-4d7b-b609-cc5fd4d55c89 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:33.507355 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f82978c-d4bf-4d7b-b609-cc5fd4d55c89 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:33.507637 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f82978c-d4bf-4d7b-b609-cc5fd4d55c89 None admin] PATH_INFO: `/v3/users/c717178de72f493788554ab410b527de` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:33.510572 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-080f87e8-0628-4d79-98a2-e7793a49ed79 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:33.515263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f82978c-d4bf-4d7b-b609-cc5fd4d55c89 None admin] RBAC: Authorizing `identity:get_user(user_id=c717178de72f493788554ab410b527de)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:33.516762 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f82978c-d4bf-4d7b-b609-cc5fd4d55c89 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:33.518465 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 84/165] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:30:33 2022] GET /identity/v3/users/c717178de72f493788554ab410b527de => generated 275 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:33.519257 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f82978c-d4bf-4d7b-b609-cc5fd4d55c89 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:33.545280 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:33.556881 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251f2506-7905-4128-a94f-784652f6c1e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:33.559269 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251f2506-7905-4128-a94f-784652f6c1e0 None admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:33.559269 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251f2506-7905-4128-a94f-784652f6c1e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:33.560744 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251f2506-7905-4128-a94f-784652f6c1e0 None admin] PATH_INFO: `/v3/projects/6e28083c55ff4290af640f011fd2466f/users/d287c1589ecb4e1caa5ee5f36033e00e/roles/12f06a0e43d345aba46a51c26c012ce3` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:33.563737 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-251f2506-7905-4128-a94f-784652f6c1e0 None admin] RBAC: Authorizing `identity:create_grant(project_id=6e28083c55ff4290af640f011fd2466f, user_id=d287c1589ecb4e1caa5ee5f36033e00e, role_id=12f06a0e43d345aba46a51c26c012ce3)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:33.569005 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-251f2506-7905-4128-a94f-784652f6c1e0 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:33.603968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 82/166] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Oct 28 09:30:33 2022] PUT /identity/v3/projects/6e28083c55ff4290af640f011fd2466f/users/d287c1589ecb4e1caa5ee5f36033e00e/roles/12f06a0e43d345aba46a51c26c012ce3 => generated 0 bytes in 361 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:30:33.603968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251f2506-7905-4128-a94f-784652f6c1e0 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:33.792398 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f82978c-d4bf-4d7b-b609-cc5fd4d55c89 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:33.833564 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:33.860541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251f2506-7905-4128-a94f-784652f6c1e0 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:33.909225 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6ec85d-d723-47ea-81f1-33598204c0c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:33.909852 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6ec85d-d723-47ea-81f1-33598204c0c7 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:33.910131 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6ec85d-d723-47ea-81f1-33598204c0c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:33.910418 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6ec85d-d723-47ea-81f1-33598204c0c7 None admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:33.911118 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c6ec85d-d723-47ea-81f1-33598204c0c7 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:33.911423 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c6ec85d-d723-47ea-81f1-33598204c0c7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:33.915104 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e19d91-c5c9-4707-8430-c8bb2b0eddd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:33.915771 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e19d91-c5c9-4707-8430-c8bb2b0eddd2 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:33.916047 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e19d91-c5c9-4707-8430-c8bb2b0eddd2 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:33.916317 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e19d91-c5c9-4707-8430-c8bb2b0eddd2 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:33.918344 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c6ec85d-d723-47ea-81f1-33598204c0c7 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:33.941211 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 85/167] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:30:33 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 422 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:33.942711 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6ec85d-d723-47ea-81f1-33598204c0c7 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:33.943968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e19d91-c5c9-4707-8430-c8bb2b0eddd2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:30:33.955957 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-d0e19d91-c5c9-4707-8430-c8bb2b0eddd2 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 09:30:33.955957 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:30:33.955957 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:30:33.955957 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:30:33.955957 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:30:33.955957 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:30:33.955957 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:30:33.955957 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:30:33.955957 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:30:33.955957 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:30:33.955957 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:30:33.955957 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:30:33.955957 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:30:33.955957 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:30:33.955957 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:30:33.955957 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:30:33.956996 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:30:33.956996 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:30:33.956996 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:30:33.956996 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:30:33.956996 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:30:33.956996 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:30:33.956996 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:30:33.956996 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:30:33.956996 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:30:33.956996 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:30:33.956996 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:30:33.956996 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:30:33.956996 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:30:33.956996 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:30:33.956996 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:30:33.956996 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:30:33.957792 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:30:33.957792 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:30:33.957792 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:30:33.957792 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:30:33.957792 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:30:33.957792 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:30:33.957792 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:30:33.957792 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:30:33.957792 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:30:33.957792 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:30:33.957792 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 09:30:33.957792 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:30:33.957792 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 83/168] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:30:33 2022] GET /identity/v3/roles/member => generated 84 bytes in 354 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:30:33.959654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e19d91-c5c9-4707-8430-c8bb2b0eddd2 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:34.098962 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:34.194688 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6ec85d-d723-47ea-81f1-33598204c0c7 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:34.215542 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e19d91-c5c9-4707-8430-c8bb2b0eddd2 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:34.278629 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6aee2d-2d95-437e-b544-1614fbf53206 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:34.279257 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6aee2d-2d95-437e-b544-1614fbf53206 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:34.279535 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6aee2d-2d95-437e-b544-1614fbf53206 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:34.279806 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6aee2d-2d95-437e-b544-1614fbf53206 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:34.297862 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943f955c-37a8-4133-9c28-88d43d6fc658 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:34.298534 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943f955c-37a8-4133-9c28-88d43d6fc658 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:34.298825 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943f955c-37a8-4133-9c28-88d43d6fc658 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:34.299105 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943f955c-37a8-4133-9c28-88d43d6fc658 None admin] PATH_INFO: `/v3/users/c717178de72f493788554ab410b527de` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:34.300529 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-943f955c-37a8-4133-9c28-88d43d6fc658 None admin] RBAC: Authorizing `identity:get_user(user_id=c717178de72f493788554ab410b527de)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:34.301988 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-943f955c-37a8-4133-9c28-88d43d6fc658 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:34.303799 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 84/169] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:30:33 2022] GET /identity/v3/users/c717178de72f493788554ab410b527de => generated 275 bytes in 346 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:34.310209 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943f955c-37a8-4133-9c28-88d43d6fc658 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:34.400898 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:34.450039 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c6aee2d-2d95-437e-b544-1614fbf53206 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:34.454552 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c6aee2d-2d95-437e-b544-1614fbf53206 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:34.540398 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943f955c-37a8-4133-9c28-88d43d6fc658 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:34.598255 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:34.658567 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 86/170] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:30:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 716 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 09:30:34.659798 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6aee2d-2d95-437e-b544-1614fbf53206 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:34.660016 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2c24e5-ad2b-4a3f-afbe-be134e63a743 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:34.660679 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2c24e5-ad2b-4a3f-afbe-be134e63a743 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:34.660943 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2c24e5-ad2b-4a3f-afbe-be134e63a743 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:34.661225 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2c24e5-ad2b-4a3f-afbe-be134e63a743 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:34.661907 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed2c24e5-ad2b-4a3f-afbe-be134e63a743 None admin] RBAC: Adding query filter params (name=member) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:34.662506 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed2c24e5-ad2b-4a3f-afbe-be134e63a743 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:34.663718 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed2c24e5-ad2b-4a3f-afbe-be134e63a743 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:34.690587 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 85/171] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Oct 28 09:30:34 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 386 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:34.692047 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3904e383-c45e-4ce7-bb76-1b6eea8d0ab4 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:34.692342 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3904e383-c45e-4ce7-bb76-1b6eea8d0ab4 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:34.692622 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3904e383-c45e-4ce7-bb76-1b6eea8d0ab4 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:34.694290 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 86/172] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:30:34.695091 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3904e383-c45e-4ce7-bb76-1b6eea8d0ab4 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:34.814785 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:34.875638 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6aee2d-2d95-437e-b544-1614fbf53206 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:34.947805 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3904e383-c45e-4ce7-bb76-1b6eea8d0ab4 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:34.953039 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-141d78f3-0407-4555-aa4e-505c10ae38d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:34.959391 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141d78f3-0407-4555-aa4e-505c10ae38d1 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:34.959680 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141d78f3-0407-4555-aa4e-505c10ae38d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:34.959947 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141d78f3-0407-4555-aa4e-505c10ae38d1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:34.996090 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078a7671-e4df-4f20-a5dd-a1723bff873e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:34.998121 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078a7671-e4df-4f20-a5dd-a1723bff873e None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:34.998448 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078a7671-e4df-4f20-a5dd-a1723bff873e None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:34.998750 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078a7671-e4df-4f20-a5dd-a1723bff873e None admin] PATH_INFO: `/v3/projects/6e28083c55ff4290af640f011fd2466f` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:35.000115 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-078a7671-e4df-4f20-a5dd-a1723bff873e None admin] RBAC: Authorizing `identity:get_project(project_id=6e28083c55ff4290af640f011fd2466f)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:35.004263 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-078a7671-e4df-4f20-a5dd-a1723bff873e None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:35.006201 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 87/173] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:30:34 2022] GET /identity/v3/projects/6e28083c55ff4290af640f011fd2466f => generated 298 bytes in 311 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:35.006823 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078a7671-e4df-4f20-a5dd-a1723bff873e None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:35.064617 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-141d78f3-0407-4555-aa4e-505c10ae38d1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:35.067975 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-141d78f3-0407-4555-aa4e-505c10ae38d1 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:35.113564 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:35.189122 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:35.204794 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078a7671-e4df-4f20-a5dd-a1723bff873e None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:35.223652 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 87/174] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:30:34 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 564 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 09:30:35.227022 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683a8d2d-c06a-403f-8eda-685f563f5787 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:35.227318 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683a8d2d-c06a-403f-8eda-685f563f5787 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:35.227597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683a8d2d-c06a-403f-8eda-685f563f5787 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:35.247650 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc251318-974c-4cb6-b7c1-c837249838e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:35.248297 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc251318-974c-4cb6-b7c1-c837249838e7 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:35.248582 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc251318-974c-4cb6-b7c1-c837249838e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:35.248863 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc251318-974c-4cb6-b7c1-c837249838e7 None admin] PATH_INFO: `/v3/users/c717178de72f493788554ab410b527de` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:35.250013 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc251318-974c-4cb6-b7c1-c837249838e7 None admin] RBAC: Authorizing `identity:get_user(user_id=c717178de72f493788554ab410b527de)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:35.251743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc251318-974c-4cb6-b7c1-c837249838e7 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:35.252347 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-683a8d2d-c06a-403f-8eda-685f563f5787 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:35.253798 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 88/175] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:30:35 2022] GET /identity/v3/users/c717178de72f493788554ab410b527de => generated 275 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:35.254124 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-683a8d2d-c06a-403f-8eda-685f563f5787 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:35.255112 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b2b33f-8b34-4600-ab7f-b342c334bed6 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:35.255401 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b2b33f-8b34-4600-ab7f-b342c334bed6 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:35.255679 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b2b33f-8b34-4600-ab7f-b342c334bed6 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:35.256897 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 89/176] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:30:35.261994 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b2b33f-8b34-4600-ab7f-b342c334bed6 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:35.345904 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 88/177] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:30:35.347800 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3e8bc2-ddd3-45b9-a4a2-8298708218d5 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:35.348310 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3e8bc2-ddd3-45b9-a4a2-8298708218d5 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:35.349556 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f3e8bc2-ddd3-45b9-a4a2-8298708218d5 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:35.355667 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 89/178] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:35 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:30:35.356888 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f3e8bc2-ddd3-45b9-a4a2-8298708218d5 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:35.393879 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b2b33f-8b34-4600-ab7f-b342c334bed6 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:35.428057 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:35.458658 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ac0fdb-5dc3-412e-ae79-54d013fe1aa9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:35.459314 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ac0fdb-5dc3-412e-ae79-54d013fe1aa9 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:35.459597 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ac0fdb-5dc3-412e-ae79-54d013fe1aa9 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:35.459870 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ac0fdb-5dc3-412e-ae79-54d013fe1aa9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:35.461240 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ac0fdb-5dc3-412e-ae79-54d013fe1aa9 None admin] RBAC: Adding query filter params (role.id=8943f289c9fe4feba2695b9c90cf3d51, scope.project.id=6e28083c55ff4290af640f011fd2466f, user.id=c717178de72f493788554ab410b527de) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:35.461734 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ac0fdb-5dc3-412e-ae79-54d013fe1aa9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:35.463991 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ac0fdb-5dc3-412e-ae79-54d013fe1aa9 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:35.478407 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 90/179] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:30:35 2022] GET /identity/v3/role_assignments?user.id=c717178de72f493788554ab410b527de&scope.project.id=6e28083c55ff4290af640f011fd2466f&role.id=8943f289c9fe4feba2695b9c90cf3d51 => generated 608 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:35.479615 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ac0fdb-5dc3-412e-ae79-54d013fe1aa9 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:35.562524 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f3e8bc2-ddd3-45b9-a4a2-8298708218d5 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:35.590462 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:35.640934 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b782b968-5d6c-400d-acb7-a06973773e10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:35.641599 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b782b968-5d6c-400d-acb7-a06973773e10 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:35.641906 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b782b968-5d6c-400d-acb7-a06973773e10 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:35.642191 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b782b968-5d6c-400d-acb7-a06973773e10 None admin] PATH_INFO: `/v3/projects/6e28083c55ff4290af640f011fd2466f` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:35.647136 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b782b968-5d6c-400d-acb7-a06973773e10 None admin] RBAC: Authorizing `identity:get_project(project_id=6e28083c55ff4290af640f011fd2466f)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:35.648721 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b782b968-5d6c-400d-acb7-a06973773e10 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:35.659784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 90/180] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:30:35 2022] GET /identity/v3/projects/6e28083c55ff4290af640f011fd2466f => generated 298 bytes in 303 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:35.661201 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c67b93b-0cb1-4494-ae52-357262f8bfbe None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:35.661491 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c67b93b-0cb1-4494-ae52-357262f8bfbe None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:35.661805 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c67b93b-0cb1-4494-ae52-357262f8bfbe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:35.701540 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-2c67b93b-0cb1-4494-ae52-357262f8bfbe None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:35.706435 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-2c67b93b-0cb1-4494-ae52-357262f8bfbe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:35.720183 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ac0fdb-5dc3-412e-ae79-54d013fe1aa9 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:35.793837 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b63d3a-96ef-44a0-8fa5-3df6cbd46c07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:35.794669 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b63d3a-96ef-44a0-8fa5-3df6cbd46c07 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:35.795055 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b63d3a-96ef-44a0-8fa5-3df6cbd46c07 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:35.796299 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b63d3a-96ef-44a0-8fa5-3df6cbd46c07 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:35.861635 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 91/181] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:30:35.872105 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1862ce9b-0a08-4271-a688-d7eae63ce698 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:35.872487 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1862ce9b-0a08-4271-a688-d7eae63ce698 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:35.872647 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1862ce9b-0a08-4271-a688-d7eae63ce698 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:35.875159 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:35.898116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b63d3a-96ef-44a0-8fa5-3df6cbd46c07 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:35.900523 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b63d3a-96ef-44a0-8fa5-3df6cbd46c07 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:35.918852 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-1862ce9b-0a08-4271-a688-d7eae63ce698 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:35.922102 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-1862ce9b-0a08-4271-a688-d7eae63ce698 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:36.043156 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 92/182] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:30:36.043795 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1862ce9b-0a08-4271-a688-d7eae63ce698 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:36.049629 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 91/183] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:30:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 570 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 09:30:36.050523 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b63d3a-96ef-44a0-8fa5-3df6cbd46c07 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:36.121173 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:36.223623 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:36.248500 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1862ce9b-0a08-4271-a688-d7eae63ce698 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:36.248805 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b63d3a-96ef-44a0-8fa5-3df6cbd46c07 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:36.304260 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0be5dbf-0bf9-4dd2-aecc-af1f8d18c722 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:36.305076 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0be5dbf-0bf9-4dd2-aecc-af1f8d18c722 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:36.305389 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0be5dbf-0bf9-4dd2-aecc-af1f8d18c722 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:36.310427 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0be5dbf-0bf9-4dd2-aecc-af1f8d18c722 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:36.310427 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0be5dbf-0bf9-4dd2-aecc-af1f8d18c722 None admin] RBAC: Adding query filter params (role.id=d93b4c94b590457199c3d301f947b0fb, scope.project.id=6e28083c55ff4290af640f011fd2466f, user.id=c717178de72f493788554ab410b527de) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:36.310427 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0be5dbf-0bf9-4dd2-aecc-af1f8d18c722 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:36.310427 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0be5dbf-0bf9-4dd2-aecc-af1f8d18c722 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:36.310804 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a4ecf9-22f6-4722-adb7-c48c185c9cc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:36.310804 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a4ecf9-22f6-4722-adb7-c48c185c9cc2 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:36.310804 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a4ecf9-22f6-4722-adb7-c48c185c9cc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:36.310804 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a4ecf9-22f6-4722-adb7-c48c185c9cc2 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:36.315553 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 93/184] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:30:36 2022] GET /identity/v3/role_assignments?user.id=c717178de72f493788554ab410b527de&scope.project.id=6e28083c55ff4290af640f011fd2466f&role.id=d93b4c94b590457199c3d301f947b0fb => generated 608 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:36.316718 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0be5dbf-0bf9-4dd2-aecc-af1f8d18c722 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:36.332185 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a4ecf9-22f6-4722-adb7-c48c185c9cc2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:30:36.339138 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-f3a4ecf9-22f6-4722-adb7-c48c185c9cc2 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:30:36.339138 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:30:36.339138 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:30:36.339138 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:30:36.339138 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:30:36.339138 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:30:36.339138 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:30:36.339138 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:30:36.339138 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:30:36.339138 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:30:36.339138 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:30:36.339138 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:30:36.339138 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:30:36.339138 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:30:36.339138 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:30:36.339138 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:30:36.340007 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:30:36.340007 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:30:36.340007 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:30:36.340007 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:30:36.340007 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:30:36.340007 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:30:36.340007 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:30:36.340007 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:30:36.340007 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:30:36.340007 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:30:36.340007 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:30:36.340007 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:30:36.340007 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:30:36.340007 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:30:36.340007 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:30:36.340007 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:30:36.340793 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:30:36.340793 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:30:36.340793 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:30:36.340793 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:30:36.340793 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:30:36.340793 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:30:36.340793 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:30:36.340793 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:30:36.340793 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:30:36.340793 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:30:36.340793 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:30:36.340793 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:30:36.346688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 92/185] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Oct 28 09:30:36 2022] GET /identity/v3/roles/admin => generated 83 bytes in 290 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:30:36.348560 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a4ecf9-22f6-4722-adb7-c48c185c9cc2 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:36.471581 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:36.479263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0be5dbf-0bf9-4dd2-aecc-af1f8d18c722 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:36.539762 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a4ecf9-22f6-4722-adb7-c48c185c9cc2 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:36.543828 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8489e6-9a33-4e8d-913a-5fc3644ef488 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:36.544977 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8489e6-9a33-4e8d-913a-5fc3644ef488 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:36.545727 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8489e6-9a33-4e8d-913a-5fc3644ef488 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:36.546514 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8489e6-9a33-4e8d-913a-5fc3644ef488 None admin] PATH_INFO: `/v3/projects/f075fea87a3c4191a9ad2ed506c8f6a3` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:36.550124 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff8489e6-9a33-4e8d-913a-5fc3644ef488 None admin] RBAC: Authorizing `identity:get_project(project_id=f075fea87a3c4191a9ad2ed506c8f6a3)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:36.553449 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff8489e6-9a33-4e8d-913a-5fc3644ef488 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:36.558829 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 94/186] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:30:36 2022] GET /identity/v3/projects/f075fea87a3c4191a9ad2ed506c8f6a3 => generated 312 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:36.561650 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8489e6-9a33-4e8d-913a-5fc3644ef488 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:36.593417 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82215caf-8663-4fd5-adbb-c7f5855a0ea5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:36.594310 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82215caf-8663-4fd5-adbb-c7f5855a0ea5 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:36.594779 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82215caf-8663-4fd5-adbb-c7f5855a0ea5 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:36.595060 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82215caf-8663-4fd5-adbb-c7f5855a0ea5 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:36.596341 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82215caf-8663-4fd5-adbb-c7f5855a0ea5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:36.598439 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82215caf-8663-4fd5-adbb-c7f5855a0ea5 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:36.602745 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 93/187] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:30:36 2022] GET /identity/v3/domains/default => generated 199 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:36.603593 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82215caf-8663-4fd5-adbb-c7f5855a0ea5 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:36.619968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:36.695573 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8489e6-9a33-4e8d-913a-5fc3644ef488 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:36.733635 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82215caf-8663-4fd5-adbb-c7f5855a0ea5 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:36.736056 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c73e8f3-b61f-47e9-a915-720947f78fb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:36.738775 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c73e8f3-b61f-47e9-a915-720947f78fb7 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:36.739057 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c73e8f3-b61f-47e9-a915-720947f78fb7 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:36.739485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c73e8f3-b61f-47e9-a915-720947f78fb7 None admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:36.740182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c73e8f3-b61f-47e9-a915-720947f78fb7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:36.740492 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c73e8f3-b61f-47e9-a915-720947f78fb7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:36.741894 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c73e8f3-b61f-47e9-a915-720947f78fb7 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:36.751578 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 95/188] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:30:36 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:36.753176 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628ebd48-234d-4eda-8908-2a9e32cc33a7 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:36.753444 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628ebd48-234d-4eda-8908-2a9e32cc33a7 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:36.753721 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628ebd48-234d-4eda-8908-2a9e32cc33a7 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:36.755015 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 96/189] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:30:36.755716 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628ebd48-234d-4eda-8908-2a9e32cc33a7 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:36.777887 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af4eaa9-1d06-42f0-abe5-5c87bd18ba5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:36.778575 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af4eaa9-1d06-42f0-abe5-5c87bd18ba5d None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:36.778855 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af4eaa9-1d06-42f0-abe5-5c87bd18ba5d None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:36.779131 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af4eaa9-1d06-42f0-abe5-5c87bd18ba5d None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:36.791506 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af4eaa9-1d06-42f0-abe5-5c87bd18ba5d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:30:36.792020 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:36.797412 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-6af4eaa9-1d06-42f0-abe5-5c87bd18ba5d None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 09:30:36.797412 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:30:36.797412 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:30:36.797412 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:30:36.797412 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:30:36.797412 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:30:36.797412 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:30:36.797412 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:30:36.797412 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:30:36.797412 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:30:36.797412 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:30:36.797412 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:30:36.797412 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:30:36.797412 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:30:36.797412 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:30:36.797412 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:30:36.798382 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:30:36.798382 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:30:36.798382 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:30:36.798382 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:30:36.798382 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:30:36.798382 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:30:36.798382 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:30:36.798382 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:30:36.798382 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:30:36.798382 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:30:36.798382 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:30:36.798382 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:30:36.798382 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:30:36.798382 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:30:36.798382 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:30:36.798382 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:30:36.799267 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:30:36.799267 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:30:36.799267 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:30:36.799267 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:30:36.799267 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:30:36.799267 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:30:36.799267 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:30:36.799267 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:30:36.799267 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:30:36.799267 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:30:36.799267 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 09:30:36.799267 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:30:36.799267 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 94/190] 10.222.0.25 () {56 vars in 1114 bytes} [Fri Oct 28 09:30:36 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:30:36.799913 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af4eaa9-1d06-42f0-abe5-5c87bd18ba5d None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:36.878511 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:36.879947 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628ebd48-234d-4eda-8908-2a9e32cc33a7 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:36.913259 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af4eaa9-1d06-42f0-abe5-5c87bd18ba5d None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:36.921323 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2afb2d53-02b2-455e-a909-ccb7af793d7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:36.922039 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2afb2d53-02b2-455e-a909-ccb7af793d7c None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:36.922388 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2afb2d53-02b2-455e-a909-ccb7af793d7c None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:36.922908 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2afb2d53-02b2-455e-a909-ccb7af793d7c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:36.924125 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2afb2d53-02b2-455e-a909-ccb7af793d7c None admin] RBAC: Adding query filter params (role.id=8943f289c9fe4feba2695b9c90cf3d51, scope.project.id=f075fea87a3c4191a9ad2ed506c8f6a3, user.id=c717178de72f493788554ab410b527de) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:36.924125 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2afb2d53-02b2-455e-a909-ccb7af793d7c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:36.925861 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2afb2d53-02b2-455e-a909-ccb7af793d7c None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:36.933203 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 97/191] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:30:36 2022] GET /identity/v3/role_assignments?user.id=c717178de72f493788554ab410b527de&scope.project.id=f075fea87a3c4191a9ad2ed506c8f6a3&role.id=8943f289c9fe4feba2695b9c90cf3d51 => generated 608 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:36.934286 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2afb2d53-02b2-455e-a909-ccb7af793d7c None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:36.960960 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf3ac91-5722-47fc-94f3-c42135b05afb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:36.961983 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf3ac91-5722-47fc-94f3-c42135b05afb None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:36.962266 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf3ac91-5722-47fc-94f3-c42135b05afb None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:36.962573 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf3ac91-5722-47fc-94f3-c42135b05afb None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:36.963717 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cf3ac91-5722-47fc-94f3-c42135b05afb None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:36.964756 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cf3ac91-5722-47fc-94f3-c42135b05afb None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:36.994455 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 95/192] 10.222.0.25 () {60 vars in 1132 bytes} [Fri Oct 28 09:30:36 2022] POST /identity/v3/users => generated 297 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:30:37.002654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aceb397-e697-4c2e-a3c5-724a7bf64169 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:37.002993 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aceb397-e697-4c2e-a3c5-724a7bf64169 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:37.003321 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aceb397-e697-4c2e-a3c5-724a7bf64169 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:37.028053 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-3aceb397-e697-4c2e-a3c5-724a7bf64169 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:37.029724 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-3aceb397-e697-4c2e-a3c5-724a7bf64169 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:37.110981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:37.127868 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 96/193] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:30:37.134650 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2afb2d53-02b2-455e-a909-ccb7af793d7c None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:37.135744 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aceb397-e697-4c2e-a3c5-724a7bf64169 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:37.183489 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce95ded-7e29-4866-899f-80db45cf7410 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:37.184396 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce95ded-7e29-4866-899f-80db45cf7410 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:37.185136 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce95ded-7e29-4866-899f-80db45cf7410 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:37.185547 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce95ded-7e29-4866-899f-80db45cf7410 None admin] PATH_INFO: `/v3/users/daccdd71141b41668e500f437943cb45` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:37.211930 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ce95ded-7e29-4866-899f-80db45cf7410 None admin] RBAC: Authorizing `identity:get_user(user_id=daccdd71141b41668e500f437943cb45)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:37.213703 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ce95ded-7e29-4866-899f-80db45cf7410 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:37.217774 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 98/194] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:30:36 2022] GET /identity/v3/users/daccdd71141b41668e500f437943cb45 => generated 283 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:37.223416 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce95ded-7e29-4866-899f-80db45cf7410 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:37.232559 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:37.312292 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aceb397-e697-4c2e-a3c5-724a7bf64169 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:37.384396 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c53f0b-a0b5-4e6d-9204-1e4f9ab9bef0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:37.386231 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c53f0b-a0b5-4e6d-9204-1e4f9ab9bef0 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:37.386231 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c53f0b-a0b5-4e6d-9204-1e4f9ab9bef0 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:37.386231 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c53f0b-a0b5-4e6d-9204-1e4f9ab9bef0 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:37.387034 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c53f0b-a0b5-4e6d-9204-1e4f9ab9bef0 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:37.387409 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c53f0b-a0b5-4e6d-9204-1e4f9ab9bef0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:37.389517 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c53f0b-a0b5-4e6d-9204-1e4f9ab9bef0 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:37.404823 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 97/195] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:30:37 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:37.410656 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce95ded-7e29-4866-899f-80db45cf7410 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:37.412805 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c53f0b-a0b5-4e6d-9204-1e4f9ab9bef0 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:37.483944 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177ffd3a-5644-49c9-8175-5f5d3f35db30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:37.484910 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177ffd3a-5644-49c9-8175-5f5d3f35db30 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:37.485192 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177ffd3a-5644-49c9-8175-5f5d3f35db30 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:37.485461 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177ffd3a-5644-49c9-8175-5f5d3f35db30 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:37.517476 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:37.604302 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c53f0b-a0b5-4e6d-9204-1e4f9ab9bef0 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:37.629792 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:37.632277 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-177ffd3a-5644-49c9-8175-5f5d3f35db30 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:37.635246 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-177ffd3a-5644-49c9-8175-5f5d3f35db30 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:37.663704 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc43f738-87e5-455f-94a2-9ebea67a0726 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:37.664248 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc43f738-87e5-455f-94a2-9ebea67a0726 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:37.664537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc43f738-87e5-455f-94a2-9ebea67a0726 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:37.664818 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc43f738-87e5-455f-94a2-9ebea67a0726 None admin] PATH_INFO: `/v3/users/daccdd71141b41668e500f437943cb45` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:37.667116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc43f738-87e5-455f-94a2-9ebea67a0726 None admin] RBAC: Authorizing `identity:get_user(user_id=daccdd71141b41668e500f437943cb45)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:37.668792 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc43f738-87e5-455f-94a2-9ebea67a0726 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:37.676243 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 98/196] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:30:37 2022] GET /identity/v3/users/daccdd71141b41668e500f437943cb45 => generated 283 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:37.677201 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc43f738-87e5-455f-94a2-9ebea67a0726 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:37.786240 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 99/197] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:30:37 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 563 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 28 09:30:37.787501 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177ffd3a-5644-49c9-8175-5f5d3f35db30 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:37.869087 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc43f738-87e5-455f-94a2-9ebea67a0726 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:37.900806 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:37.922870 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a682bed1-e1aa-4763-adba-4eed6365edff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:37.924171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a682bed1-e1aa-4763-adba-4eed6365edff None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:37.924633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a682bed1-e1aa-4763-adba-4eed6365edff None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:37.925086 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a682bed1-e1aa-4763-adba-4eed6365edff None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:37.942793 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a682bed1-e1aa-4763-adba-4eed6365edff None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:30:37.950003 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-a682bed1-e1aa-4763-adba-4eed6365edff None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 09:30:37.950003 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:30:37.950003 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:30:37.950003 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:30:37.950003 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:30:37.950003 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:30:37.950003 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:30:37.950003 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:30:37.950003 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:30:37.950003 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:30:37.950003 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:30:37.950003 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:30:37.950003 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:30:37.950003 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:30:37.950003 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:30:37.950003 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:30:37.951037 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:30:37.951037 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:30:37.951037 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:30:37.951037 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:30:37.951037 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:30:37.951037 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:30:37.951037 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:30:37.951037 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:30:37.951037 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:30:37.951037 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:30:37.951037 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:30:37.951037 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:30:37.951037 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:30:37.951037 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:30:37.951037 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:30:37.951037 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:30:37.951833 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:30:37.951833 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:30:37.951833 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:30:37.951833 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:30:37.951833 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:30:37.951833 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:30:37.951833 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:30:37.951833 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:30:37.951833 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:30:37.951833 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:30:37.951833 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 09:30:37.951833 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:30:37.954246 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 99/198] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:30:37 2022] GET /identity/v3/roles/reader => generated 84 bytes in 277 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:30:37.955243 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a682bed1-e1aa-4763-adba-4eed6365edff None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:38.005367 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-177ffd3a-5644-49c9-8175-5f5d3f35db30 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:38.025336 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:38.075461 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c45abf60-c532-4822-890e-bf509f0fa2e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:38.077060 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45abf60-c532-4822-890e-bf509f0fa2e9 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:38.077060 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45abf60-c532-4822-890e-bf509f0fa2e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:38.077060 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c45abf60-c532-4822-890e-bf509f0fa2e9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:38.112323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a682bed1-e1aa-4763-adba-4eed6365edff None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:38.174656 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86840d3b-2942-4909-a6d0-3f102477b092 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:38.175246 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86840d3b-2942-4909-a6d0-3f102477b092 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:38.175549 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86840d3b-2942-4909-a6d0-3f102477b092 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:38.176078 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86840d3b-2942-4909-a6d0-3f102477b092 None admin] PATH_INFO: `/v3/projects/5960af42464f4e83b60ec52264d04b70` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:38.180296 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86840d3b-2942-4909-a6d0-3f102477b092 None admin] RBAC: Authorizing `identity:get_project(project_id=5960af42464f4e83b60ec52264d04b70)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:38.182612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86840d3b-2942-4909-a6d0-3f102477b092 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:38.184130 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 100/199] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:30:37 2022] GET /identity/v3/projects/5960af42464f4e83b60ec52264d04b70 => generated 302 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:38.184910 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86840d3b-2942-4909-a6d0-3f102477b092 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:38.189889 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c45abf60-c532-4822-890e-bf509f0fa2e9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:38.192350 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c45abf60-c532-4822-890e-bf509f0fa2e9 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:38.306454 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:38.337617 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86840d3b-2942-4909-a6d0-3f102477b092 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:38.358236 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:38.393202 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3fa350d-8612-4deb-8964-59e06996f793 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:38.394212 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fa350d-8612-4deb-8964-59e06996f793 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:38.394755 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fa350d-8612-4deb-8964-59e06996f793 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:38.395243 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fa350d-8612-4deb-8964-59e06996f793 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:38.396305 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3fa350d-8612-4deb-8964-59e06996f793 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:38.396814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3fa350d-8612-4deb-8964-59e06996f793 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:38.399406 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3fa350d-8612-4deb-8964-59e06996f793 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:38.416038 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 101/200] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Oct 28 09:30:38 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:38.417337 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3fa350d-8612-4deb-8964-59e06996f793 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:38.420463 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 100/201] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:30:37 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 633 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 09:30:38.432241 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c45abf60-c532-4822-890e-bf509f0fa2e9 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:38.556984 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3fa350d-8612-4deb-8964-59e06996f793 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:38.605543 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37615c76-f029-455c-8e9d-2cc02711f9f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:38.607717 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37615c76-f029-455c-8e9d-2cc02711f9f8 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:38.607717 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37615c76-f029-455c-8e9d-2cc02711f9f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:38.607717 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37615c76-f029-455c-8e9d-2cc02711f9f8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:38.608838 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37615c76-f029-455c-8e9d-2cc02711f9f8 None admin] RBAC: Adding query filter params (role.id=e13d798a98ea4c3dbb87a236ec683d32, scope.project.id=5960af42464f4e83b60ec52264d04b70, user.id=daccdd71141b41668e500f437943cb45) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:38.609155 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37615c76-f029-455c-8e9d-2cc02711f9f8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:38.610912 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37615c76-f029-455c-8e9d-2cc02711f9f8 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:38.619093 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 102/202] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:30:38 2022] GET /identity/v3/role_assignments?user.id=daccdd71141b41668e500f437943cb45&scope.project.id=5960af42464f4e83b60ec52264d04b70&role.id=e13d798a98ea4c3dbb87a236ec683d32 => generated 259 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:38.620216 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37615c76-f029-455c-8e9d-2cc02711f9f8 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:38.626679 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c45abf60-c532-4822-890e-bf509f0fa2e9 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:38.711415 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b8b985-64ea-4604-bc0a-03f5151f271b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:38.712378 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b8b985-64ea-4604-bc0a-03f5151f271b None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:38.712846 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b8b985-64ea-4604-bc0a-03f5151f271b None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:38.713393 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b8b985-64ea-4604-bc0a-03f5151f271b None admin] PATH_INFO: `/v3/users/d287c1589ecb4e1caa5ee5f36033e00e` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:38.715047 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56b8b985-64ea-4604-bc0a-03f5151f271b None admin] RBAC: Authorizing `identity:get_user(user_id=d287c1589ecb4e1caa5ee5f36033e00e)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:38.720577 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:38.723474 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56b8b985-64ea-4604-bc0a-03f5151f271b None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:38.725541 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 101/203] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:30:38 2022] GET /identity/v3/users/d287c1589ecb4e1caa5ee5f36033e00e => generated 289 bytes in 299 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:38.731872 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b8b985-64ea-4604-bc0a-03f5151f271b None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:38.786313 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:38.858130 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37615c76-f029-455c-8e9d-2cc02711f9f8 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:38.909396 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ab1516-e576-4b7e-b2c1-11ee119079b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:38.910805 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ab1516-e576-4b7e-b2c1-11ee119079b0 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:38.911270 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ab1516-e576-4b7e-b2c1-11ee119079b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:38.911745 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ab1516-e576-4b7e-b2c1-11ee119079b0 None admin] PATH_INFO: `/v3/projects/5960af42464f4e83b60ec52264d04b70` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:38.913116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ab1516-e576-4b7e-b2c1-11ee119079b0 None admin] RBAC: Authorizing `identity:get_project(project_id=5960af42464f4e83b60ec52264d04b70)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:38.915303 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ab1516-e576-4b7e-b2c1-11ee119079b0 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:38.917263 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 103/204] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:30:38 2022] GET /identity/v3/projects/5960af42464f4e83b60ec52264d04b70 => generated 302 bytes in 298 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:38.924909 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ab1516-e576-4b7e-b2c1-11ee119079b0 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:39.020558 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b8b985-64ea-4604-bc0a-03f5151f271b None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:39.077578 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ab1516-e576-4b7e-b2c1-11ee119079b0 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:39.083765 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:39.118612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-101c9f5a-6765-47fe-9512-97410e7518ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:39.119242 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101c9f5a-6765-47fe-9512-97410e7518ec None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:39.119527 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101c9f5a-6765-47fe-9512-97410e7518ec None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:39.119805 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-101c9f5a-6765-47fe-9512-97410e7518ec None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:39.120544 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-101c9f5a-6765-47fe-9512-97410e7518ec None admin] RBAC: Adding query filter params (role.id=d93b4c94b590457199c3d301f947b0fb, scope.project.id=5960af42464f4e83b60ec52264d04b70, user.id=daccdd71141b41668e500f437943cb45) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:39.120864 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-101c9f5a-6765-47fe-9512-97410e7518ec None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:39.122333 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-101c9f5a-6765-47fe-9512-97410e7518ec None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:39.129918 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 104/205] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:30:38 2022] GET /identity/v3/role_assignments?user.id=daccdd71141b41668e500f437943cb45&scope.project.id=5960af42464f4e83b60ec52264d04b70&role.id=d93b4c94b590457199c3d301f947b0fb => generated 259 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:39.151221 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8db4b6f-306e-4886-a147-4d73216fc2f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:39.151873 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8db4b6f-306e-4886-a147-4d73216fc2f4 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:39.152156 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8db4b6f-306e-4886-a147-4d73216fc2f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:39.152429 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8db4b6f-306e-4886-a147-4d73216fc2f4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:39.285317 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8db4b6f-306e-4886-a147-4d73216fc2f4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:39.286857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8db4b6f-306e-4886-a147-4d73216fc2f4 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:39.339420 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:39.427677 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 102/206] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:30:38 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 696 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 28 09:30:39.433097 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-101c9f5a-6765-47fe-9512-97410e7518ec None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:39.568082 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-101c9f5a-6765-47fe-9512-97410e7518ec None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:39.606890 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b16360-d418-4c9a-b538-2e0e0e0e6408 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:39.607601 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b16360-d418-4c9a-b538-2e0e0e0e6408 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:39.607897 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b16360-d418-4c9a-b538-2e0e0e0e6408 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:39.608175 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b16360-d418-4c9a-b538-2e0e0e0e6408 None admin] PATH_INFO: `/v3/projects/6e28083c55ff4290af640f011fd2466f` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:39.609336 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b16360-d418-4c9a-b538-2e0e0e0e6408 None admin] RBAC: Authorizing `identity:get_project(project_id=6e28083c55ff4290af640f011fd2466f)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:39.610814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b16360-d418-4c9a-b538-2e0e0e0e6408 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:39.612813 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 105/207] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:30:39 2022] GET /identity/v3/projects/6e28083c55ff4290af640f011fd2466f => generated 298 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:39.620380 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8db4b6f-306e-4886-a147-4d73216fc2f4 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:39.811594 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8db4b6f-306e-4886-a147-4d73216fc2f4 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:39.841573 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:39.883404 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-032ead79-ae84-4b77-be44-61a1aee33431 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:39.884049 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032ead79-ae84-4b77-be44-61a1aee33431 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:39.884335 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032ead79-ae84-4b77-be44-61a1aee33431 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:39.884617 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032ead79-ae84-4b77-be44-61a1aee33431 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:39.885361 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-032ead79-ae84-4b77-be44-61a1aee33431 None admin] RBAC: Adding query filter params (role.id=12f06a0e43d345aba46a51c26c012ce3, scope.project.id=6e28083c55ff4290af640f011fd2466f, user.id=d287c1589ecb4e1caa5ee5f36033e00e) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:39.885675 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-032ead79-ae84-4b77-be44-61a1aee33431 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:39.887232 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-032ead79-ae84-4b77-be44-61a1aee33431 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:39.897375 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 103/208] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:30:39 2022] GET /identity/v3/role_assignments?user.id=d287c1589ecb4e1caa5ee5f36033e00e&scope.project.id=6e28083c55ff4290af640f011fd2466f&role.id=12f06a0e43d345aba46a51c26c012ce3 => generated 608 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:40.009057 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e145bc-693a-47b4-bcff-a057dee5234e None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:40.009578 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e145bc-693a-47b4-bcff-a057dee5234e None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:40.009662 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e145bc-693a-47b4-bcff-a057dee5234e None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:40.011154 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 106/209] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:30:40.019250 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f81d60-2aa9-4137-a87e-3baaa5b48a17 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:40.019566 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f81d60-2aa9-4137-a87e-3baaa5b48a17 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:40.019813 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f81d60-2aa9-4137-a87e-3baaa5b48a17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:40.044108 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3a36ca-c72f-4a62-bcc6-4a66121e93e8 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:40.044108 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3a36ca-c72f-4a62-bcc6-4a66121e93e8 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:40.044108 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3a36ca-c72f-4a62-bcc6-4a66121e93e8 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:40.044108 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 107/210] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:30:40.051920 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a6bd4a6-9f9d-4413-826a-db561986778f None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:40.052223 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a6bd4a6-9f9d-4413-826a-db561986778f None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:40.052478 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a6bd4a6-9f9d-4413-826a-db561986778f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:40.055925 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-09f81d60-2aa9-4137-a87e-3baaa5b48a17 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:40.057960 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-09f81d60-2aa9-4137-a87e-3baaa5b48a17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:40.103957 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-1a6bd4a6-9f9d-4413-826a-db561986778f None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:40.105374 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-1a6bd4a6-9f9d-4413-826a-db561986778f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:40.188475 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:40.209026 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 104/211] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:30:40.245782 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 108/212] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:30:40.532564 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09f81d60-2aa9-4137-a87e-3baaa5b48a17 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:40.590428 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a6bd4a6-9f9d-4413-826a-db561986778f None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:40.679220 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09f81d60-2aa9-4137-a87e-3baaa5b48a17 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:40.698288 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:40.730278 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a6bd4a6-9f9d-4413-826a-db561986778f None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:40.731638 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4ab976-4c8e-48a3-940f-b7d875ef52f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:40.732577 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4ab976-4c8e-48a3-940f-b7d875ef52f8 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:40.733063 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4ab976-4c8e-48a3-940f-b7d875ef52f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:40.733611 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac4ab976-4c8e-48a3-940f-b7d875ef52f8 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:40.750467 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac4ab976-4c8e-48a3-940f-b7d875ef52f8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:30:40.757200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-ac4ab976-4c8e-48a3-940f-b7d875ef52f8 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 09:30:40.757200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:30:40.757200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:30:40.757200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:30:40.757200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:30:40.757200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:30:40.757200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:30:40.757200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:30:40.757200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:30:40.757200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:30:40.757200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:30:40.757200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:30:40.757200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:30:40.757200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:30:40.757200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:30:40.757200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:30:40.758205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:30:40.758205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:30:40.758205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:30:40.758205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:30:40.758205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:30:40.758205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:30:40.758205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:30:40.758205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:30:40.758205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:30:40.758205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:30:40.758205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:30:40.758205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:30:40.758205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:30:40.758205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:30:40.758205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:30:40.758205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:30:40.759026 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:30:40.759026 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:30:40.759026 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:30:40.759026 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:30:40.759026 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:30:40.759026 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:30:40.759026 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:30:40.759026 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:30:40.759026 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:30:40.759026 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:30:40.759026 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 09:30:40.759026 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:30:40.765075 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 105/213] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:30:40 2022] GET /identity/v3/roles/member => generated 84 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:30:40.774656 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4ab976-4c8e-48a3-940f-b7d875ef52f8 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:40.781744 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a45c05f3-c956-40ca-bc1a-1bc9902271c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:40.782774 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45c05f3-c956-40ca-bc1a-1bc9902271c5 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:40.783415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45c05f3-c956-40ca-bc1a-1bc9902271c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:40.783415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45c05f3-c956-40ca-bc1a-1bc9902271c5 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:40.784536 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a45c05f3-c956-40ca-bc1a-1bc9902271c5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:40.786019 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a45c05f3-c956-40ca-bc1a-1bc9902271c5 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:40.789360 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 109/214] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:30:40 2022] GET /identity/v3/domains/default => generated 199 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:30:40.798641 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a45c05f3-c956-40ca-bc1a-1bc9902271c5 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:40.895176 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:40.941413 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac4ab976-4c8e-48a3-940f-b7d875ef52f8 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:40.995476 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22cbeca-450a-4ffb-aa36-286b0c6b222c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:40.996882 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22cbeca-450a-4ffb-aa36-286b0c6b222c None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:40.998013 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22cbeca-450a-4ffb-aa36-286b0c6b222c None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:40.998773 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22cbeca-450a-4ffb-aa36-286b0c6b222c None admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:41.000645 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f22cbeca-450a-4ffb-aa36-286b0c6b222c None admin] RBAC: Adding query filter params (name=member) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:41.001211 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f22cbeca-450a-4ffb-aa36-286b0c6b222c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:41.004037 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f22cbeca-450a-4ffb-aa36-286b0c6b222c None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:41.010810 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a45c05f3-c956-40ca-bc1a-1bc9902271c5 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:41.028201 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 106/215] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Oct 28 09:30:40 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:41.035394 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22cbeca-450a-4ffb-aa36-286b0c6b222c None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:41.087577 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460e79fc-9f4f-4c93-901e-70eeb0f8812f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:41.088506 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460e79fc-9f4f-4c93-901e-70eeb0f8812f None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:41.089093 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460e79fc-9f4f-4c93-901e-70eeb0f8812f None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:41.089537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460e79fc-9f4f-4c93-901e-70eeb0f8812f None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:41.090554 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-460e79fc-9f4f-4c93-901e-70eeb0f8812f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:41.092213 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-460e79fc-9f4f-4c93-901e-70eeb0f8812f None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:41.123252 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:41.132753 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 110/216] 10.222.0.25 () {60 vars in 1132 bytes} [Fri Oct 28 09:30:40 2022] POST /identity/v3/users => generated 297 bytes in 330 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:30:41.217427 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f22cbeca-450a-4ffb-aa36-286b0c6b222c None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:41.261248 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56f8676-9661-49fb-9e18-bcc3d8cb98b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:41.262190 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56f8676-9661-49fb-9e18-bcc3d8cb98b8 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:41.262646 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56f8676-9661-49fb-9e18-bcc3d8cb98b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:41.263099 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b56f8676-9661-49fb-9e18-bcc3d8cb98b8 None admin] PATH_INFO: `/v3/users/1ac90379b2454e689b29dc7e52c529a7` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:41.267944 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61f62cb-6d06-42d9-a1a9-d292087c2b92 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:41.268552 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61f62cb-6d06-42d9-a1a9-d292087c2b92 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:41.268967 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61f62cb-6d06-42d9-a1a9-d292087c2b92 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:41.270559 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 111/217] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:30:41.277054 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2218b1af-aad2-49cd-b7ce-af58599c04c3 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:41.277594 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2218b1af-aad2-49cd-b7ce-af58599c04c3 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:41.278011 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2218b1af-aad2-49cd-b7ce-af58599c04c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:41.298935 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b56f8676-9661-49fb-9e18-bcc3d8cb98b8 None admin] RBAC: Authorizing `identity:get_user(user_id=1ac90379b2454e689b29dc7e52c529a7)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:41.300844 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b56f8676-9661-49fb-9e18-bcc3d8cb98b8 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:41.301244 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-2218b1af-aad2-49cd-b7ce-af58599c04c3 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:41.302896 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-2218b1af-aad2-49cd-b7ce-af58599c04c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:41.303322 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 107/218] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:30:41 2022] GET /identity/v3/users/1ac90379b2454e689b29dc7e52c529a7 => generated 297 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:41.310209 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56f8676-9661-49fb-9e18-bcc3d8cb98b8 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:41.413243 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 112/219] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 09:30:41.483968 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:41.495874 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b56f8676-9661-49fb-9e18-bcc3d8cb98b8 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:41.544552 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-022dd02b-8ecc-4489-aa98-26b406c46d96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:41.545553 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022dd02b-8ecc-4489-aa98-26b406c46d96 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:41.546041 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022dd02b-8ecc-4489-aa98-26b406c46d96 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:41.546552 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-022dd02b-8ecc-4489-aa98-26b406c46d96 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:41.649768 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-022dd02b-8ecc-4489-aa98-26b406c46d96 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:41.650754 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-022dd02b-8ecc-4489-aa98-26b406c46d96 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:41.742849 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2218b1af-aad2-49cd-b7ce-af58599c04c3 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:41.773262 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:41.787461 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:41.790613 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 108/220] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:30:41 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 481 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 09:30:41.791579 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-022dd02b-8ecc-4489-aa98-26b406c46d96 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:41.882013 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2218b1af-aad2-49cd-b7ce-af58599c04c3 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:41.941655 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569c1cbd-e2da-450f-b6a6-49e0521ddf2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:41.942484 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569c1cbd-e2da-450f-b6a6-49e0521ddf2f None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:41.943601 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569c1cbd-e2da-450f-b6a6-49e0521ddf2f None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:41.943983 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569c1cbd-e2da-450f-b6a6-49e0521ddf2f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:41.963319 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-569c1cbd-e2da-450f-b6a6-49e0521ddf2f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:30:41.981633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-569c1cbd-e2da-450f-b6a6-49e0521ddf2f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:30:41.981633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:30:41.981633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:30:41.981633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:30:41.981633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:30:41.981633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:30:41.981633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:30:41.981633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:30:41.981633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:30:41.981633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:30:41.981633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:30:41.981633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:30:41.981633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:30:41.981633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:30:41.981633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:30:41.981633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:30:41.982643 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:30:41.982643 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:30:41.982643 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:30:41.982643 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:30:41.982643 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:30:41.982643 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:30:41.982643 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:30:41.982643 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:30:41.982643 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:30:41.982643 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:30:41.982643 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:30:41.982643 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:30:41.982643 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:30:41.982643 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:30:41.982643 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:30:41.982643 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:30:41.983432 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:30:41.983432 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:30:41.983432 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:30:41.983432 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:30:41.983432 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:30:41.983432 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:30:41.983432 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:30:41.983432 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:30:41.983432 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:30:41.983432 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:30:41.983432 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:30:41.983432 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:30:41.985221 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 113/221] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Oct 28 09:30:41 2022] GET /identity/v3/roles/admin => generated 83 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:30:41.996120 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569c1cbd-e2da-450f-b6a6-49e0521ddf2f None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:42.060438 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-022dd02b-8ecc-4489-aa98-26b406c46d96 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:42.107785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:42.134283 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569c1cbd-e2da-450f-b6a6-49e0521ddf2f None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:42.160652 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b4e108-e8c3-4203-bc8a-2cd3a96b1b40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:42.161369 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b4e108-e8c3-4203-bc8a-2cd3a96b1b40 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:42.161863 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b4e108-e8c3-4203-bc8a-2cd3a96b1b40 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:42.162155 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b4e108-e8c3-4203-bc8a-2cd3a96b1b40 None admin] PATH_INFO: `/v3/projects/5960af42464f4e83b60ec52264d04b70` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:42.163410 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b4e108-e8c3-4203-bc8a-2cd3a96b1b40 None admin] RBAC: Authorizing `identity:get_project(project_id=5960af42464f4e83b60ec52264d04b70)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:42.167279 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b4e108-e8c3-4203-bc8a-2cd3a96b1b40 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:42.169050 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 109/222] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:30:41 2022] GET /identity/v3/projects/5960af42464f4e83b60ec52264d04b70 => generated 302 bytes in 378 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:42.173181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e094211e-37a9-4181-8dc0-14f0ee025470 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:42.173469 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e094211e-37a9-4181-8dc0-14f0ee025470 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:42.173751 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e094211e-37a9-4181-8dc0-14f0ee025470 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:42.177740 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 110/223] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:42 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:30:42.178681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e094211e-37a9-4181-8dc0-14f0ee025470 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:42.188292 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e88688-6036-42c8-ac92-5b19203b1d3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:42.188933 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e88688-6036-42c8-ac92-5b19203b1d3f None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:42.189284 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e88688-6036-42c8-ac92-5b19203b1d3f None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:42.189489 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e88688-6036-42c8-ac92-5b19203b1d3f None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:42.190183 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e88688-6036-42c8-ac92-5b19203b1d3f None admin] RBAC: Adding query filter params (name=admin) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:42.191603 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e88688-6036-42c8-ac92-5b19203b1d3f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:42.191979 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e88688-6036-42c8-ac92-5b19203b1d3f None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:42.201966 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 114/224] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:30:41 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:42.203192 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe56575-471b-4858-9192-5c78b054ea9b None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:42.203499 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe56575-471b-4858-9192-5c78b054ea9b None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:42.205421 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe56575-471b-4858-9192-5c78b054ea9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:42.229853 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-7fe56575-471b-4858-9192-5c78b054ea9b None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:42.231108 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7fe56575-471b-4858-9192-5c78b054ea9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:42.305420 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:42.359499 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e094211e-37a9-4181-8dc0-14f0ee025470 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:42.373185 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 115/225] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:30:42.375655 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe56575-471b-4858-9192-5c78b054ea9b None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:42.455927 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f380dbb7-f971-4c5e-9374-e073b9ceeb71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:42.456531 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f380dbb7-f971-4c5e-9374-e073b9ceeb71 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:42.456817 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f380dbb7-f971-4c5e-9374-e073b9ceeb71 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:42.457095 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f380dbb7-f971-4c5e-9374-e073b9ceeb71 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:42.458427 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f380dbb7-f971-4c5e-9374-e073b9ceeb71 None admin] RBAC: Adding query filter params (role.id=8943f289c9fe4feba2695b9c90cf3d51, scope.project.id=5960af42464f4e83b60ec52264d04b70, user.id=1ac90379b2454e689b29dc7e52c529a7) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:42.458427 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f380dbb7-f971-4c5e-9374-e073b9ceeb71 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:42.459914 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f380dbb7-f971-4c5e-9374-e073b9ceeb71 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:42.475769 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 111/226] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:30:42 2022] GET /identity/v3/role_assignments?user.id=1ac90379b2454e689b29dc7e52c529a7&scope.project.id=5960af42464f4e83b60ec52264d04b70&role.id=8943f289c9fe4feba2695b9c90cf3d51 => generated 259 bytes in 297 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:42.521083 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:42.559276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe56575-471b-4858-9192-5c78b054ea9b None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:42.641719 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70bb553-4eff-4f40-9d95-b2303610c852 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:42.642496 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70bb553-4eff-4f40-9d95-b2303610c852 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:42.642880 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70bb553-4eff-4f40-9d95-b2303610c852 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:42.643166 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70bb553-4eff-4f40-9d95-b2303610c852 None admin] PATH_INFO: `/v3/users/daccdd71141b41668e500f437943cb45` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:42.644514 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70bb553-4eff-4f40-9d95-b2303610c852 None admin] RBAC: Authorizing `identity:get_user(user_id=daccdd71141b41668e500f437943cb45)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:42.646034 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70bb553-4eff-4f40-9d95-b2303610c852 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:42.649862 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 116/227] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:30:42 2022] GET /identity/v3/users/daccdd71141b41668e500f437943cb45 => generated 283 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:42.656688 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f380dbb7-f971-4c5e-9374-e073b9ceeb71 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:42.783187 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f380dbb7-f971-4c5e-9374-e073b9ceeb71 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:42.820889 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b385bc-a19f-4c98-9e43-e4ee3a9743d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:42.821547 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b385bc-a19f-4c98-9e43-e4ee3a9743d5 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:42.821841 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b385bc-a19f-4c98-9e43-e4ee3a9743d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:42.826604 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:42.827143 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b385bc-a19f-4c98-9e43-e4ee3a9743d5 None admin] PATH_INFO: `/v3/projects/5960af42464f4e83b60ec52264d04b70` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:42.828882 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b385bc-a19f-4c98-9e43-e4ee3a9743d5 None admin] RBAC: Authorizing `identity:get_project(project_id=5960af42464f4e83b60ec52264d04b70)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:42.830784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b385bc-a19f-4c98-9e43-e4ee3a9743d5 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:42.833653 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 112/228] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:30:42 2022] GET /identity/v3/projects/5960af42464f4e83b60ec52264d04b70 => generated 302 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:42.841338 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70bb553-4eff-4f40-9d95-b2303610c852 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:42.892545 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b385bc-a19f-4c98-9e43-e4ee3a9743d5 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:43.025918 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70bb553-4eff-4f40-9d95-b2303610c852 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:43.113744 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b385bc-a19f-4c98-9e43-e4ee3a9743d5 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:43.128182 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f59d1e-ca28-4489-8892-c4516c3aedd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:43.129453 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f59d1e-ca28-4489-8892-c4516c3aedd4 None admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:43.129881 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f59d1e-ca28-4489-8892-c4516c3aedd4 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:43.130292 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f59d1e-ca28-4489-8892-c4516c3aedd4 None admin] PATH_INFO: `/v3/projects/5960af42464f4e83b60ec52264d04b70/users/daccdd71141b41668e500f437943cb45/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:43.132501 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29f59d1e-ca28-4489-8892-c4516c3aedd4 None admin] RBAC: Authorizing `identity:create_grant(project_id=5960af42464f4e83b60ec52264d04b70, user_id=daccdd71141b41668e500f437943cb45, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:43.134721 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29f59d1e-ca28-4489-8892-c4516c3aedd4 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:43.170382 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b19d1358-a75b-4d92-8242-909198ec35c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:43.171229 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19d1358-a75b-4d92-8242-909198ec35c5 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:43.171754 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19d1358-a75b-4d92-8242-909198ec35c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:43.173113 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19d1358-a75b-4d92-8242-909198ec35c5 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:43.186655 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 117/229] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Oct 28 09:30:42 2022] PUT /identity/v3/projects/5960af42464f4e83b60ec52264d04b70/users/daccdd71141b41668e500f437943cb45/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 336 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:30:43.203981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b19d1358-a75b-4d92-8242-909198ec35c5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:30:43.226422 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-b19d1358-a75b-4d92-8242-909198ec35c5 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 09:30:43.226422 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:30:43.226422 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:30:43.226422 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:30:43.226422 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:30:43.226422 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:30:43.226422 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:30:43.226422 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:30:43.226422 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:30:43.226422 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:30:43.226422 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:30:43.226422 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:30:43.226422 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:30:43.226422 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:30:43.226422 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:30:43.226422 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:30:43.227957 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:30:43.227957 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:30:43.227957 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:30:43.227957 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:30:43.227957 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:30:43.227957 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:30:43.227957 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:30:43.227957 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:30:43.227957 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:30:43.227957 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:30:43.227957 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:30:43.227957 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:30:43.227957 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:30:43.227957 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:30:43.227957 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:30:43.227957 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:30:43.228745 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:30:43.228745 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:30:43.228745 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:30:43.228745 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:30:43.228745 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:30:43.228745 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:30:43.228745 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:30:43.228745 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:30:43.228745 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:30:43.228745 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:30:43.228745 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 09:30:43.228745 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:30:43.228745 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 113/230] 10.222.0.25 () {56 vars in 1114 bytes} [Fri Oct 28 09:30:42 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 332 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:30:43.233065 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f59d1e-ca28-4489-8892-c4516c3aedd4 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:43.242424 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:43.361690 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f59d1e-ca28-4489-8892-c4516c3aedd4 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:43.415562 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6c34cb-e212-4239-84c0-2ed18e38ef89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:43.416316 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6c34cb-e212-4239-84c0-2ed18e38ef89 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:43.416603 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6c34cb-e212-4239-84c0-2ed18e38ef89 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:43.416884 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6c34cb-e212-4239-84c0-2ed18e38ef89 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:43.417573 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c6c34cb-e212-4239-84c0-2ed18e38ef89 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:43.417881 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c6c34cb-e212-4239-84c0-2ed18e38ef89 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:43.419305 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c6c34cb-e212-4239-84c0-2ed18e38ef89 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:43.435046 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 118/231] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:30:43 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:43.442424 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b19d1358-a75b-4d92-8242-909198ec35c5 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:43.565734 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:43.630425 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b19d1358-a75b-4d92-8242-909198ec35c5 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:43.686841 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5756c2a0-8dfc-4fcf-8741-63eaa37d9f69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:43.687519 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5756c2a0-8dfc-4fcf-8741-63eaa37d9f69 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:43.687802 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5756c2a0-8dfc-4fcf-8741-63eaa37d9f69 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:43.688073 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5756c2a0-8dfc-4fcf-8741-63eaa37d9f69 None admin] PATH_INFO: `/v3/users/daccdd71141b41668e500f437943cb45` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:43.691286 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5756c2a0-8dfc-4fcf-8741-63eaa37d9f69 None admin] RBAC: Authorizing `identity:get_user(user_id=daccdd71141b41668e500f437943cb45)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:43.692783 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5756c2a0-8dfc-4fcf-8741-63eaa37d9f69 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:43.695142 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 114/232] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:30:43 2022] GET /identity/v3/users/daccdd71141b41668e500f437943cb45 => generated 283 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:43.702070 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6c34cb-e212-4239-84c0-2ed18e38ef89 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:43.864829 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:43.938553 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6c34cb-e212-4239-84c0-2ed18e38ef89 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:43.995557 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b80e279-8f7d-4909-af10-2a6c371eb8a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:43.996221 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b80e279-8f7d-4909-af10-2a6c371eb8a1 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:43.996507 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b80e279-8f7d-4909-af10-2a6c371eb8a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:43.996790 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b80e279-8f7d-4909-af10-2a6c371eb8a1 None admin] PATH_INFO: `/v3/projects/5960af42464f4e83b60ec52264d04b70` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:44.000876 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b80e279-8f7d-4909-af10-2a6c371eb8a1 None admin] RBAC: Authorizing `identity:get_project(project_id=5960af42464f4e83b60ec52264d04b70)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:44.000876 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b80e279-8f7d-4909-af10-2a6c371eb8a1 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:44.001440 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 119/233] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:30:43 2022] GET /identity/v3/projects/5960af42464f4e83b60ec52264d04b70 => generated 302 bytes in 300 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:44.007164 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5756c2a0-8dfc-4fcf-8741-63eaa37d9f69 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:44.235226 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5756c2a0-8dfc-4fcf-8741-63eaa37d9f69 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:44.282316 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589b7a7b-e8ac-4a2c-9d47-5c96e8d0c1cb None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:44.282877 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:44.283436 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589b7a7b-e8ac-4a2c-9d47-5c96e8d0c1cb None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:44.284139 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-589b7a7b-e8ac-4a2c-9d47-5c96e8d0c1cb None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:44.286859 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 120/234] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:44 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:30:44.296728 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21fbf64-6e07-4924-b16b-c96c999130ee None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:44.297359 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21fbf64-6e07-4924-b16b-c96c999130ee None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:44.297851 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21fbf64-6e07-4924-b16b-c96c999130ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:44.322230 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53faf68b-7cd5-4cec-bb4a-c243a699a7e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:44.322967 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53faf68b-7cd5-4cec-bb4a-c243a699a7e1 None admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:44.323253 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53faf68b-7cd5-4cec-bb4a-c243a699a7e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:44.323529 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53faf68b-7cd5-4cec-bb4a-c243a699a7e1 None admin] PATH_INFO: `/v3/projects/5960af42464f4e83b60ec52264d04b70/users/daccdd71141b41668e500f437943cb45/roles/d93b4c94b590457199c3d301f947b0fb` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:44.325821 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53faf68b-7cd5-4cec-bb4a-c243a699a7e1 None admin] RBAC: Authorizing `identity:create_grant(project_id=5960af42464f4e83b60ec52264d04b70, user_id=daccdd71141b41668e500f437943cb45, role_id=d93b4c94b590457199c3d301f947b0fb)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:44.326758 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-a21fbf64-6e07-4924-b16b-c96c999130ee None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:44.327353 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53faf68b-7cd5-4cec-bb4a-c243a699a7e1 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:44.328421 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a21fbf64-6e07-4924-b16b-c96c999130ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:44.366803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 115/235] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Oct 28 09:30:44 2022] PUT /identity/v3/projects/5960af42464f4e83b60ec52264d04b70/users/daccdd71141b41668e500f437943cb45/roles/d93b4c94b590457199c3d301f947b0fb => generated 0 bytes in 360 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:30:44.464491 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 121/236] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 09:30:44.502221 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3c45be-b761-4a23-85fa-feb2f6ab878a None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:44.502474 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3c45be-b761-4a23-85fa-feb2f6ab878a None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:44.502755 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3c45be-b761-4a23-85fa-feb2f6ab878a None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:44.504158 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 116/237] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:30:44.520496 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5a7616-a838-4b29-8f18-ee8661e19b32 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:44.521180 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5a7616-a838-4b29-8f18-ee8661e19b32 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:44.521659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5a7616-a838-4b29-8f18-ee8661e19b32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:44.550970 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-ec5a7616-a838-4b29-8f18-ee8661e19b32 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:44.552639 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ec5a7616-a838-4b29-8f18-ee8661e19b32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:44.651011 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:44.659288 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 122/238] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:30:44.921930 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e3c45be-b761-4a23-85fa-feb2f6ab878a None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:44.968402 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5a7616-a838-4b29-8f18-ee8661e19b32 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:45.184693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e3c45be-b761-4a23-85fa-feb2f6ab878a None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:45.244615 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5a7616-a838-4b29-8f18-ee8661e19b32 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:45.303410 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5d627c-cff7-458c-8509-f6d212a2eb82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:45.303410 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5d627c-cff7-458c-8509-f6d212a2eb82 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:45.303410 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5d627c-cff7-458c-8509-f6d212a2eb82 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:45.304074 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5d627c-cff7-458c-8509-f6d212a2eb82 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:45.323952 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e5d627c-cff7-458c-8509-f6d212a2eb82 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:30:45.329109 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-6e5d627c-cff7-458c-8509-f6d212a2eb82 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 09:30:45.329109 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:30:45.329109 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:30:45.329109 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:30:45.329109 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:30:45.329109 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:30:45.329109 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:30:45.329109 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:30:45.329109 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:30:45.329109 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:30:45.329109 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:30:45.329109 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:30:45.329109 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:30:45.329109 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:30:45.329109 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:30:45.329109 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:30:45.329961 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:30:45.329961 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:30:45.329961 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:30:45.329961 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:30:45.329961 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:30:45.329961 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:30:45.329961 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:30:45.329961 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:30:45.329961 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:30:45.329961 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:30:45.329961 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:30:45.329961 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:30:45.329961 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:30:45.329961 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:30:45.329961 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:30:45.329961 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:30:45.330784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:30:45.330784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:30:45.330784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:30:45.330784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:30:45.330784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:30:45.330784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:30:45.330784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:30:45.330784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:30:45.330784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:30:45.330784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:30:45.330784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 09:30:45.330784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:30:45.345448 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 117/239] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:30:44 2022] GET /identity/v3/roles/reader => generated 84 bytes in 424 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:30:45.349089 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5d627c-cff7-458c-8509-f6d212a2eb82 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:45.373214 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89f6a30-b5dd-4387-b0c5-bb5c5d49bfc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:45.374521 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89f6a30-b5dd-4387-b0c5-bb5c5d49bfc0 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:45.374991 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89f6a30-b5dd-4387-b0c5-bb5c5d49bfc0 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:45.375443 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89f6a30-b5dd-4387-b0c5-bb5c5d49bfc0 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:45.381803 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89f6a30-b5dd-4387-b0c5-bb5c5d49bfc0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:45.384055 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89f6a30-b5dd-4387-b0c5-bb5c5d49bfc0 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:45.386295 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 123/240] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:30:44 2022] GET /identity/v3/domains/default => generated 199 bytes in 419 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:45.392691 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89f6a30-b5dd-4387-b0c5-bb5c5d49bfc0 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:45.467281 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:45.604158 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5d627c-cff7-458c-8509-f6d212a2eb82 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:45.612055 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:45.693856 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6453a3f1-4c79-4732-b56f-3d4675c43bb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:45.694746 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6453a3f1-4c79-4732-b56f-3d4675c43bb4 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:45.695171 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6453a3f1-4c79-4732-b56f-3d4675c43bb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:45.695620 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6453a3f1-4c79-4732-b56f-3d4675c43bb4 None admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:45.696447 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6453a3f1-4c79-4732-b56f-3d4675c43bb4 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:45.696909 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6453a3f1-4c79-4732-b56f-3d4675c43bb4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:45.698448 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6453a3f1-4c79-4732-b56f-3d4675c43bb4 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:45.707373 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89f6a30-b5dd-4387-b0c5-bb5c5d49bfc0 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:45.724510 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 118/241] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Oct 28 09:30:45 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 376 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:45.726002 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a86cef-e387-4852-aebd-ef0537537001 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:45.726513 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a86cef-e387-4852-aebd-ef0537537001 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:45.726969 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a86cef-e387-4852-aebd-ef0537537001 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:45.728409 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 119/242] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:30:45.734150 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a86cef-e387-4852-aebd-ef0537537001 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:45.784852 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd7c00d-c4e3-448c-b26c-d4c83d18a6d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:45.786535 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd7c00d-c4e3-448c-b26c-d4c83d18a6d2 None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:45.786960 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd7c00d-c4e3-448c-b26c-d4c83d18a6d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:45.787377 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd7c00d-c4e3-448c-b26c-d4c83d18a6d2 None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:45.788176 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd7c00d-c4e3-448c-b26c-d4c83d18a6d2 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:45.789807 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd7c00d-c4e3-448c-b26c-d4c83d18a6d2 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:45.829584 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 124/243] 10.222.0.25 () {60 vars in 1132 bytes} [Fri Oct 28 09:30:45 2022] POST /identity/v3/users => generated 293 bytes in 437 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:30:45.831425 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6152d803-a4a3-4cf4-a942-5c1554625c4f None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:45.834064 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6152d803-a4a3-4cf4-a942-5c1554625c4f None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:45.837717 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6152d803-a4a3-4cf4-a942-5c1554625c4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:45.869444 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-6152d803-a4a3-4cf4-a942-5c1554625c4f None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:45.871900 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-6152d803-a4a3-4cf4-a942-5c1554625c4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:45.957502 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a86cef-e387-4852-aebd-ef0537537001 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:45.968328 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:46.035998 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 125/244] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:30:46.050556 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9dfdc20-ab4b-4fc0-aee6-49a0b4bb5c59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:46.052683 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9dfdc20-ab4b-4fc0-aee6-49a0b4bb5c59 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:46.053175 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9dfdc20-ab4b-4fc0-aee6-49a0b4bb5c59 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:46.053752 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9dfdc20-ab4b-4fc0-aee6-49a0b4bb5c59 None admin] PATH_INFO: `/v3/users/ac5c183a78ce4f15bb0191916824c968` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:46.094487 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9dfdc20-ab4b-4fc0-aee6-49a0b4bb5c59 None admin] RBAC: Authorizing `identity:get_user(user_id=ac5c183a78ce4f15bb0191916824c968)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:46.100041 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9dfdc20-ab4b-4fc0-aee6-49a0b4bb5c59 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:46.102930 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 120/245] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:30:45 2022] GET /identity/v3/users/ac5c183a78ce4f15bb0191916824c968 => generated 297 bytes in 369 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:46.109076 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6152d803-a4a3-4cf4-a942-5c1554625c4f None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:46.219024 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:46.244121 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a372f08-0d22-4c59-8da6-411b9698a71f None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:46.244722 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a372f08-0d22-4c59-8da6-411b9698a71f None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:46.245197 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a372f08-0d22-4c59-8da6-411b9698a71f None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:46.247075 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 121/246] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 09:30:46.251925 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b013af-83e0-4fa9-bdcd-384e14d95bfb None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:46.252455 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b013af-83e0-4fa9-bdcd-384e14d95bfb None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:46.252926 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b013af-83e0-4fa9-bdcd-384e14d95bfb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:46.295162 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-77b013af-83e0-4fa9-bdcd-384e14d95bfb None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:46.298178 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-77b013af-83e0-4fa9-bdcd-384e14d95bfb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:46.345528 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6152d803-a4a3-4cf4-a942-5c1554625c4f None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:46.430887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 122/247] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:30:46.434699 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3934fe0e-b220-479b-9560-d58c1456b7ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:46.435855 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3934fe0e-b220-479b-9560-d58c1456b7ad None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:46.436437 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3934fe0e-b220-479b-9560-d58c1456b7ad None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:46.436906 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3934fe0e-b220-479b-9560-d58c1456b7ad None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:46.504136 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b013af-83e0-4fa9-bdcd-384e14d95bfb None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:46.598433 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3934fe0e-b220-479b-9560-d58c1456b7ad None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:46.598433 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3934fe0e-b220-479b-9560-d58c1456b7ad None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:46.706829 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:46.726438 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:46.761608 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b013af-83e0-4fa9-bdcd-384e14d95bfb None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:46.786029 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 126/248] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:30:46 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 677 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 09:30:46.792390 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3934fe0e-b220-479b-9560-d58c1456b7ad None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:46.828474 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f33a084-0ebf-4dde-b7f9-337056a48800 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:46.829517 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f33a084-0ebf-4dde-b7f9-337056a48800 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:46.829992 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f33a084-0ebf-4dde-b7f9-337056a48800 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:46.830485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f33a084-0ebf-4dde-b7f9-337056a48800 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:46.842942 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f33a084-0ebf-4dde-b7f9-337056a48800 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:30:46.848492 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-2f33a084-0ebf-4dde-b7f9-337056a48800 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 09:30:46.848492 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:30:46.848492 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:30:46.848492 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:30:46.848492 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:30:46.848492 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:30:46.848492 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:30:46.848492 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:30:46.848492 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:30:46.848492 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:30:46.848492 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:30:46.848492 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:30:46.848492 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:30:46.848492 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:30:46.848492 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:30:46.848492 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:30:46.849365 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:30:46.849365 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:30:46.849365 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:30:46.849365 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:30:46.849365 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:30:46.849365 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:30:46.849365 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:30:46.849365 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:30:46.849365 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:30:46.849365 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:30:46.849365 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:30:46.849365 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:30:46.849365 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:30:46.849365 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:30:46.849365 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:30:46.849365 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:30:46.850142 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:30:46.850142 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:30:46.850142 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:30:46.850142 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:30:46.850142 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:30:46.850142 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:30:46.850142 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:30:46.850142 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:30:46.850142 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:30:46.850142 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:30:46.850142 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 09:30:46.850142 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:30:46.852586 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 123/249] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:30:46 2022] GET /identity/v3/roles/member => generated 84 bytes in 349 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:30:46.853595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f33a084-0ebf-4dde-b7f9-337056a48800 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:46.908276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3934fe0e-b220-479b-9560-d58c1456b7ad None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:46.949223 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a91d5b2-7bef-4f67-8d67-0f2977ea781e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:46.950152 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a91d5b2-7bef-4f67-8d67-0f2977ea781e None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:46.950613 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a91d5b2-7bef-4f67-8d67-0f2977ea781e None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:46.951033 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a91d5b2-7bef-4f67-8d67-0f2977ea781e None admin] PATH_INFO: `/v3/projects/5960af42464f4e83b60ec52264d04b70` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:46.952330 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a91d5b2-7bef-4f67-8d67-0f2977ea781e None admin] RBAC: Authorizing `identity:get_project(project_id=5960af42464f4e83b60ec52264d04b70)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:46.953942 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a91d5b2-7bef-4f67-8d67-0f2977ea781e None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:46.956345 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 127/250] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:30:46 2022] GET /identity/v3/projects/5960af42464f4e83b60ec52264d04b70 => generated 302 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:30:46.957681 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a91d5b2-7bef-4f67-8d67-0f2977ea781e None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:46.966095 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f33a084-0ebf-4dde-b7f9-337056a48800 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:46.977754 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:47.004748 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d7f01e-be9c-436b-b137-98db405aa690 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:47.006131 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d7f01e-be9c-436b-b137-98db405aa690 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:47.006914 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d7f01e-be9c-436b-b137-98db405aa690 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:47.007632 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d7f01e-be9c-436b-b137-98db405aa690 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:47.019804 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06d7f01e-be9c-436b-b137-98db405aa690 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:30:47.026739 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-06d7f01e-be9c-436b-b137-98db405aa690 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:30:47.026739 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:30:47.026739 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:30:47.026739 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:30:47.026739 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:30:47.026739 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:30:47.026739 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:30:47.026739 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:30:47.026739 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:30:47.026739 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:30:47.026739 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:30:47.026739 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:30:47.026739 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:30:47.026739 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:30:47.026739 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:30:47.026739 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:30:47.027627 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:30:47.027627 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:30:47.027627 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:30:47.027627 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:30:47.027627 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:30:47.027627 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:30:47.027627 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:30:47.027627 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:30:47.027627 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:30:47.027627 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:30:47.027627 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:30:47.027627 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:30:47.027627 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:30:47.027627 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:30:47.027627 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:30:47.027627 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:30:47.028398 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:30:47.028398 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:30:47.028398 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:30:47.028398 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:30:47.028398 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:30:47.028398 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:30:47.028398 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:30:47.028398 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:30:47.028398 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:30:47.028398 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:30:47.028398 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:30:47.028398 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:30:47.030709 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 124/251] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Oct 28 09:30:46 2022] GET /identity/v3/roles/admin => generated 83 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:30:47.031703 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d7f01e-be9c-436b-b137-98db405aa690 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:47.053249 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:47.068299 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a91d5b2-7bef-4f67-8d67-0f2977ea781e None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:47.116719 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc2261de-001d-481c-a8fd-04c02354e137 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:47.117316 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2261de-001d-481c-a8fd-04c02354e137 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:47.117605 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2261de-001d-481c-a8fd-04c02354e137 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:47.117887 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2261de-001d-481c-a8fd-04c02354e137 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:47.118595 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc2261de-001d-481c-a8fd-04c02354e137 None admin] RBAC: Adding query filter params (name=member) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:47.118909 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc2261de-001d-481c-a8fd-04c02354e137 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:47.120295 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc2261de-001d-481c-a8fd-04c02354e137 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:47.142411 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 128/252] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Oct 28 09:30:46 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:47.143352 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc2261de-001d-481c-a8fd-04c02354e137 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:47.222892 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d7f01e-be9c-436b-b137-98db405aa690 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:47.279949 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:47.298602 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-667cf531-9090-449a-b9ae-7f921a02a8d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:47.299231 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667cf531-9090-449a-b9ae-7f921a02a8d7 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:47.299521 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667cf531-9090-449a-b9ae-7f921a02a8d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:47.299798 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-667cf531-9090-449a-b9ae-7f921a02a8d7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:47.300540 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-667cf531-9090-449a-b9ae-7f921a02a8d7 None admin] RBAC: Adding query filter params (role.id=12f06a0e43d345aba46a51c26c012ce3, scope.project.id=5960af42464f4e83b60ec52264d04b70, user.id=ac5c183a78ce4f15bb0191916824c968) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:47.300852 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-667cf531-9090-449a-b9ae-7f921a02a8d7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:47.302322 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-667cf531-9090-449a-b9ae-7f921a02a8d7 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:47.313213 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 125/253] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:30:47 2022] GET /identity/v3/role_assignments?user.id=ac5c183a78ce4f15bb0191916824c968&scope.project.id=5960af42464f4e83b60ec52264d04b70&role.id=12f06a0e43d345aba46a51c26c012ce3 => generated 259 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:47.314053 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-667cf531-9090-449a-b9ae-7f921a02a8d7 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:47.319182 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc2261de-001d-481c-a8fd-04c02354e137 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:47.419356 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05d2122-d79d-4d40-b27c-3d697114a559 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:47.420243 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05d2122-d79d-4d40-b27c-3d697114a559 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:47.420692 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05d2122-d79d-4d40-b27c-3d697114a559 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:47.421114 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05d2122-d79d-4d40-b27c-3d697114a559 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:47.421987 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a05d2122-d79d-4d40-b27c-3d697114a559 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:47.422474 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a05d2122-d79d-4d40-b27c-3d697114a559 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:47.424167 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a05d2122-d79d-4d40-b27c-3d697114a559 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:47.425111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:47.446022 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 129/254] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:30:47 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 303 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:47.450549 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca89a69-2139-4fcb-a6b9-8c55cb2491b3 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:47.452790 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca89a69-2139-4fcb-a6b9-8c55cb2491b3 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:47.453308 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca89a69-2139-4fcb-a6b9-8c55cb2491b3 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:47.455654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 130/255] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:47 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:30:47.458064 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fca89a69-2139-4fcb-a6b9-8c55cb2491b3 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:47.476249 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-667cf531-9090-449a-b9ae-7f921a02a8d7 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:47.521235 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6bbeb86-c010-4f2c-a004-c5484a92354d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:47.522583 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6bbeb86-c010-4f2c-a004-c5484a92354d None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:47.524498 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6bbeb86-c010-4f2c-a004-c5484a92354d None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:47.525327 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6bbeb86-c010-4f2c-a004-c5484a92354d None admin] PATH_INFO: `/v3/users/1ac90379b2454e689b29dc7e52c529a7` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:47.527260 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6bbeb86-c010-4f2c-a004-c5484a92354d None admin] RBAC: Authorizing `identity:get_user(user_id=1ac90379b2454e689b29dc7e52c529a7)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:47.529058 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6bbeb86-c010-4f2c-a004-c5484a92354d None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:47.530950 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 126/256] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:30:47 2022] GET /identity/v3/users/1ac90379b2454e689b29dc7e52c529a7 => generated 297 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:47.533278 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8bf368-1e24-40ca-92e8-31aea49bbcec None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:47.533773 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8bf368-1e24-40ca-92e8-31aea49bbcec None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:47.534668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f8bf368-1e24-40ca-92e8-31aea49bbcec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:47.562017 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-9f8bf368-1e24-40ca-92e8-31aea49bbcec None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:47.564205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-9f8bf368-1e24-40ca-92e8-31aea49bbcec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:47.623681 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fca89a69-2139-4fcb-a6b9-8c55cb2491b3 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:47.690200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 127/257] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:30:47.691181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8bf368-1e24-40ca-92e8-31aea49bbcec None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:47.702928 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a57dee8-b592-42bc-b255-5981214f7a82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:47.703572 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a57dee8-b592-42bc-b255-5981214f7a82 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:47.703852 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a57dee8-b592-42bc-b255-5981214f7a82 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:47.704123 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a57dee8-b592-42bc-b255-5981214f7a82 None admin] PATH_INFO: `/v3/users/daccdd71141b41668e500f437943cb45` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:47.705453 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a57dee8-b592-42bc-b255-5981214f7a82 None admin] RBAC: Authorizing `identity:get_user(user_id=daccdd71141b41668e500f437943cb45)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:47.708840 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a57dee8-b592-42bc-b255-5981214f7a82 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:47.711941 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 131/258] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:30:47 2022] GET /identity/v3/users/daccdd71141b41668e500f437943cb45 => generated 283 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:47.717596 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a57dee8-b592-42bc-b255-5981214f7a82 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:47.736694 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:47.918796 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:48.011014 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f8bf368-1e24-40ca-92e8-31aea49bbcec None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:48.016516 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a57dee8-b592-42bc-b255-5981214f7a82 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:48.135818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac12ff8-0e9f-42ab-856b-a6ec2809b04e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:48.136481 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac12ff8-0e9f-42ab-856b-a6ec2809b04e None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:48.136768 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac12ff8-0e9f-42ab-856b-a6ec2809b04e None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:48.137047 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac12ff8-0e9f-42ab-856b-a6ec2809b04e None admin] PATH_INFO: `/v3/projects/5960af42464f4e83b60ec52264d04b70` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:48.139359 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac12ff8-0e9f-42ab-856b-a6ec2809b04e None admin] RBAC: Authorizing `identity:get_project(project_id=5960af42464f4e83b60ec52264d04b70)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:48.140829 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac12ff8-0e9f-42ab-856b-a6ec2809b04e None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:48.146692 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 128/259] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:30:47 2022] GET /identity/v3/projects/5960af42464f4e83b60ec52264d04b70 => generated 302 bytes in 456 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:48.148918 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac12ff8-0e9f-42ab-856b-a6ec2809b04e None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:48.162736 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71490009-7783-4263-8085-33547cbbf87c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:48.163372 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71490009-7783-4263-8085-33547cbbf87c None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:48.163719 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71490009-7783-4263-8085-33547cbbf87c None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:48.163936 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71490009-7783-4263-8085-33547cbbf87c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:48.322368 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71490009-7783-4263-8085-33547cbbf87c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:48.324203 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71490009-7783-4263-8085-33547cbbf87c None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:48.337690 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:48.412350 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac12ff8-0e9f-42ab-856b-a6ec2809b04e None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:48.452233 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 132/260] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:30:47 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 735 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 09:30:48.466445 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71490009-7783-4263-8085-33547cbbf87c None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:48.523126 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2562ea45-073f-4808-90f6-d84c0104f6c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:48.524345 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2562ea45-073f-4808-90f6-d84c0104f6c6 None admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:48.524635 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2562ea45-073f-4808-90f6-d84c0104f6c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:48.524919 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2562ea45-073f-4808-90f6-d84c0104f6c6 None admin] PATH_INFO: `/v3/projects/5960af42464f4e83b60ec52264d04b70/users/1ac90379b2454e689b29dc7e52c529a7/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:48.527003 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2562ea45-073f-4808-90f6-d84c0104f6c6 None admin] RBAC: Authorizing `identity:create_grant(project_id=5960af42464f4e83b60ec52264d04b70, user_id=1ac90379b2454e689b29dc7e52c529a7, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:48.528459 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2562ea45-073f-4808-90f6-d84c0104f6c6 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:48.555186 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 129/261] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Oct 28 09:30:48 2022] PUT /identity/v3/projects/5960af42464f4e83b60ec52264d04b70/users/1ac90379b2454e689b29dc7e52c529a7/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 407 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:30:48.557388 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2562ea45-073f-4808-90f6-d84c0104f6c6 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:48.565725 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:48.654131 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71490009-7783-4263-8085-33547cbbf87c None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:48.672251 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:48.738487 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cafd8b8-9472-4474-9f99-e2b59da69d8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:48.739147 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cafd8b8-9472-4474-9f99-e2b59da69d8f None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:48.739433 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cafd8b8-9472-4474-9f99-e2b59da69d8f None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:48.739707 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cafd8b8-9472-4474-9f99-e2b59da69d8f None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:48.759218 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cafd8b8-9472-4474-9f99-e2b59da69d8f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:30:48.765528 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-1cafd8b8-9472-4474-9f99-e2b59da69d8f None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 09:30:48.765528 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:30:48.765528 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:30:48.765528 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:30:48.765528 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:30:48.765528 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:30:48.765528 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:30:48.765528 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:30:48.765528 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:30:48.765528 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:30:48.765528 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:30:48.765528 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:30:48.765528 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:30:48.765528 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:30:48.765528 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:30:48.765528 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:30:48.766485 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:30:48.766485 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:30:48.766485 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:30:48.766485 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:30:48.766485 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:30:48.766485 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:30:48.766485 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:30:48.766485 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:30:48.766485 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:30:48.766485 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:30:48.766485 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:30:48.766485 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:30:48.766485 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:30:48.766485 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:30:48.766485 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:30:48.766485 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:30:48.767836 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:30:48.767836 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:30:48.767836 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:30:48.767836 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:30:48.767836 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:30:48.767836 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:30:48.767836 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:30:48.767836 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:30:48.767836 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:30:48.767836 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:30:48.767836 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 09:30:48.767836 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:30:48.767836 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 133/262] 10.222.0.25 () {56 vars in 1114 bytes} [Fri Oct 28 09:30:48 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 314 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:30:48.775508 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cafd8b8-9472-4474-9f99-e2b59da69d8f None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:48.836514 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2562ea45-073f-4808-90f6-d84c0104f6c6 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:48.918869 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cafd8b8-9472-4474-9f99-e2b59da69d8f None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:48.960926 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c3d1544-86ed-4391-96e2-498e9624794c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:48.961590 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3d1544-86ed-4391-96e2-498e9624794c None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:48.961882 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3d1544-86ed-4391-96e2-498e9624794c None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:48.962161 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3d1544-86ed-4391-96e2-498e9624794c None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:48.962880 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c3d1544-86ed-4391-96e2-498e9624794c None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:48.963192 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c3d1544-86ed-4391-96e2-498e9624794c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:48.964604 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c3d1544-86ed-4391-96e2-498e9624794c None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:48.966977 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4ef580-bf4c-4e5f-8cdf-dc4a26efc327 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:48.967636 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4ef580-bf4c-4e5f-8cdf-dc4a26efc327 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:48.967916 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4ef580-bf4c-4e5f-8cdf-dc4a26efc327 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:48.968195 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4ef580-bf4c-4e5f-8cdf-dc4a26efc327 None admin] PATH_INFO: `/v3/projects/5960af42464f4e83b60ec52264d04b70` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:48.970651 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d4ef580-bf4c-4e5f-8cdf-dc4a26efc327 None admin] RBAC: Authorizing `identity:get_project(project_id=5960af42464f4e83b60ec52264d04b70)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:48.972136 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d4ef580-bf4c-4e5f-8cdf-dc4a26efc327 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:48.973821 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 130/263] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:30:48 2022] GET /identity/v3/projects/5960af42464f4e83b60ec52264d04b70 => generated 302 bytes in 418 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:48.980939 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4ef580-bf4c-4e5f-8cdf-dc4a26efc327 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:48.985870 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 134/264] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:30:48 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:48.997436 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c3d1544-86ed-4391-96e2-498e9624794c None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:49.037522 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:49.152671 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:49.165914 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c3d1544-86ed-4391-96e2-498e9624794c None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:49.221363 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4ef580-bf4c-4e5f-8cdf-dc4a26efc327 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:49.231873 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d591a79-161b-4299-b0a9-245a2c3eef75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:49.232541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d591a79-161b-4299-b0a9-245a2c3eef75 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:49.232828 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d591a79-161b-4299-b0a9-245a2c3eef75 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:49.233110 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d591a79-161b-4299-b0a9-245a2c3eef75 None admin] PATH_INFO: `/v3/users/daccdd71141b41668e500f437943cb45` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:49.234315 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d591a79-161b-4299-b0a9-245a2c3eef75 None admin] RBAC: Authorizing `identity:get_user(user_id=daccdd71141b41668e500f437943cb45)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:49.235802 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d591a79-161b-4299-b0a9-245a2c3eef75 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:49.241865 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 135/265] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:30:48 2022] GET /identity/v3/users/daccdd71141b41668e500f437943cb45 => generated 283 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:49.244736 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d591a79-161b-4299-b0a9-245a2c3eef75 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:49.311970 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83827c33-aaff-4827-983e-e3acd3a7f0d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:49.312628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83827c33-aaff-4827-983e-e3acd3a7f0d8 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:49.312967 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83827c33-aaff-4827-983e-e3acd3a7f0d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:49.313242 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83827c33-aaff-4827-983e-e3acd3a7f0d8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:49.314049 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83827c33-aaff-4827-983e-e3acd3a7f0d8 None admin] RBAC: Adding query filter params (role.id=e13d798a98ea4c3dbb87a236ec683d32, scope.project.id=5960af42464f4e83b60ec52264d04b70, user.id=daccdd71141b41668e500f437943cb45) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:49.314341 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83827c33-aaff-4827-983e-e3acd3a7f0d8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:49.315841 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83827c33-aaff-4827-983e-e3acd3a7f0d8 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:49.336979 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 131/266] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:30:48 2022] GET /identity/v3/role_assignments?user.id=daccdd71141b41668e500f437943cb45&scope.project.id=5960af42464f4e83b60ec52264d04b70&role.id=e13d798a98ea4c3dbb87a236ec683d32 => generated 608 bytes in 357 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:49.388151 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75731e6-a4e6-40c5-bb8d-fdcb1930d8af None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:49.388495 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75731e6-a4e6-40c5-bb8d-fdcb1930d8af None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:49.388722 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75731e6-a4e6-40c5-bb8d-fdcb1930d8af None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:49.390674 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 132/267] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:30:49.413983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f34141-1c52-4565-993c-073677451d08 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:49.414331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f34141-1c52-4565-993c-073677451d08 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:49.414581 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f34141-1c52-4565-993c-073677451d08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:49.455533 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-60f34141-1c52-4565-993c-073677451d08 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:49.456777 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-60f34141-1c52-4565-993c-073677451d08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:49.458760 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:49.462572 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d591a79-161b-4299-b0a9-245a2c3eef75 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:49.561242 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410b2303-e471-4924-b144-a9a7c76121db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:49.562054 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410b2303-e471-4924-b144-a9a7c76121db None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:49.562496 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410b2303-e471-4924-b144-a9a7c76121db None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:49.562881 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410b2303-e471-4924-b144-a9a7c76121db None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:49.630984 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 133/268] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:30:49.631959 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e932be60-d8ee-4aa5-bce7-9df2530a845c None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:49.632247 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e932be60-d8ee-4aa5-bce7-9df2530a845c None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:49.632520 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e932be60-d8ee-4aa5-bce7-9df2530a845c None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:49.634725 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 134/269] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:30:49.642899 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478f3d21-f9ca-41d3-8a23-88329f2b49b3 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:49.643170 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478f3d21-f9ca-41d3-8a23-88329f2b49b3 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:49.643447 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478f3d21-f9ca-41d3-8a23-88329f2b49b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:49.674224 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410b2303-e471-4924-b144-a9a7c76121db None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:49.676039 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410b2303-e471-4924-b144-a9a7c76121db None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:49.679995 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-478f3d21-f9ca-41d3-8a23-88329f2b49b3 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:49.680677 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-478f3d21-f9ca-41d3-8a23-88329f2b49b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:49.792780 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:49.800060 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 136/270] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:30:49 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 556 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 09:30:49.806985 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410b2303-e471-4924-b144-a9a7c76121db None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:49.808625 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:49.817689 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 135/271] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:30:49.922228 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410b2303-e471-4924-b144-a9a7c76121db None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:49.958690 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f62d35-bd49-410a-8dc5-33008c8c152a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:49.959545 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f62d35-bd49-410a-8dc5-33008c8c152a None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:49.959967 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f62d35-bd49-410a-8dc5-33008c8c152a None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:49.960380 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f62d35-bd49-410a-8dc5-33008c8c152a None admin] PATH_INFO: `/v3/projects/5960af42464f4e83b60ec52264d04b70` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:49.962199 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f62d35-bd49-410a-8dc5-33008c8c152a None admin] RBAC: Authorizing `identity:get_project(project_id=5960af42464f4e83b60ec52264d04b70)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:49.963804 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f62d35-bd49-410a-8dc5-33008c8c152a None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:49.965591 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 137/272] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:30:49 2022] GET /identity/v3/projects/5960af42464f4e83b60ec52264d04b70 => generated 302 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:49.972259 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478f3d21-f9ca-41d3-8a23-88329f2b49b3 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:50.091350 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f62d35-bd49-410a-8dc5-33008c8c152a None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:50.108626 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478f3d21-f9ca-41d3-8a23-88329f2b49b3 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:50.155968 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7b500e-1be1-4fd1-af6c-b22189b778ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:50.156934 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7b500e-1be1-4fd1-af6c-b22189b778ab None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:50.157662 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7b500e-1be1-4fd1-af6c-b22189b778ab None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:50.158129 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7b500e-1be1-4fd1-af6c-b22189b778ab None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:50.159103 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e7b500e-1be1-4fd1-af6c-b22189b778ab None admin] RBAC: Adding query filter params (role.id=d93b4c94b590457199c3d301f947b0fb, scope.project.id=5960af42464f4e83b60ec52264d04b70, user.id=daccdd71141b41668e500f437943cb45) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:50.159890 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e7b500e-1be1-4fd1-af6c-b22189b778ab None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:50.161543 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e7b500e-1be1-4fd1-af6c-b22189b778ab None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:50.169284 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 136/273] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:30:49 2022] GET /identity/v3/role_assignments?user.id=daccdd71141b41668e500f437943cb45&scope.project.id=5960af42464f4e83b60ec52264d04b70&role.id=d93b4c94b590457199c3d301f947b0fb => generated 608 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:50.202587 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7b500e-1be1-4fd1-af6c-b22189b778ab None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:50.220147 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:50.261319 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f62d35-bd49-410a-8dc5-33008c8c152a None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:50.292286 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:50.300245 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f092dfc-6bed-4bb0-bfca-9a1a6da07864 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:50.300838 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f092dfc-6bed-4bb0-bfca-9a1a6da07864 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:50.301124 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f092dfc-6bed-4bb0-bfca-9a1a6da07864 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:50.301402 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f092dfc-6bed-4bb0-bfca-9a1a6da07864 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:50.313842 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f092dfc-6bed-4bb0-bfca-9a1a6da07864 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:30:50.319799 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-0f092dfc-6bed-4bb0-bfca-9a1a6da07864 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 09:30:50.319799 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:30:50.319799 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:30:50.319799 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:30:50.319799 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:30:50.319799 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:30:50.319799 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:30:50.319799 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:30:50.319799 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:30:50.319799 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:30:50.319799 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:30:50.319799 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:30:50.319799 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:30:50.319799 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:30:50.319799 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:30:50.319799 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:30:50.321343 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:30:50.321343 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:30:50.321343 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:30:50.321343 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:30:50.321343 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:30:50.321343 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:30:50.321343 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:30:50.321343 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:30:50.321343 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:30:50.321343 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:30:50.321343 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:30:50.321343 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:30:50.321343 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:30:50.321343 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:30:50.321343 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:30:50.321343 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:30:50.322132 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:30:50.322132 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:30:50.322132 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:30:50.322132 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:30:50.322132 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:30:50.322132 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:30:50.322132 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:30:50.322132 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:30:50.322132 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:30:50.322132 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:30:50.322132 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 09:30:50.322132 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:30:50.326681 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 138/274] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:30:50 2022] GET /identity/v3/roles/member => generated 84 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:30:50.328172 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca006179-00bc-44c9-a7df-b2fd87f06b78 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:50.328548 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca006179-00bc-44c9-a7df-b2fd87f06b78 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:50.328750 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca006179-00bc-44c9-a7df-b2fd87f06b78 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:50.330191 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 139/275] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:30:50.335489 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca006179-00bc-44c9-a7df-b2fd87f06b78 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:50.359547 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7b500e-1be1-4fd1-af6c-b22189b778ab None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:50.411051 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1bb1b9-31fb-47b4-9259-0154265702ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:50.413096 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1bb1b9-31fb-47b4-9259-0154265702ed None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:50.413096 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1bb1b9-31fb-47b4-9259-0154265702ed None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:50.413096 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1bb1b9-31fb-47b4-9259-0154265702ed None admin] PATH_INFO: `/v3/domains/default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:50.413368 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b1bb1b9-31fb-47b4-9259-0154265702ed None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:50.414891 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b1bb1b9-31fb-47b4-9259-0154265702ed None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:50.416798 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 137/276] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:30:50 2022] GET /identity/v3/domains/default => generated 199 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:30:50.417984 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d18da6-0497-4ee4-84c1-257126c52e1c None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:50.418270 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d18da6-0497-4ee4-84c1-257126c52e1c None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:50.418579 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d18da6-0497-4ee4-84c1-257126c52e1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:50.439039 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-61d18da6-0497-4ee4-84c1-257126c52e1c None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:50.440257 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-61d18da6-0497-4ee4-84c1-257126c52e1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:50.470817 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca006179-00bc-44c9-a7df-b2fd87f06b78 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:50.499236 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:50.509584 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef99d4c6-6d15-40a1-8e56-80b2f05aff7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:50.510539 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef99d4c6-6d15-40a1-8e56-80b2f05aff7c None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:50.510647 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef99d4c6-6d15-40a1-8e56-80b2f05aff7c None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:50.510940 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef99d4c6-6d15-40a1-8e56-80b2f05aff7c None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:50.511608 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef99d4c6-6d15-40a1-8e56-80b2f05aff7c None admin] RBAC: Adding query filter params (name=member) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:50.511914 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef99d4c6-6d15-40a1-8e56-80b2f05aff7c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:50.513595 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef99d4c6-6d15-40a1-8e56-80b2f05aff7c None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:50.523282 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 140/277] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Oct 28 09:30:50 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:50.523882 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef99d4c6-6d15-40a1-8e56-80b2f05aff7c None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:50.533788 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 138/278] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:30:50.534713 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d18da6-0497-4ee4-84c1-257126c52e1c None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:50.598930 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:50.661065 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef99d4c6-6d15-40a1-8e56-80b2f05aff7c None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:50.665732 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d18da6-0497-4ee4-84c1-257126c52e1c None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:50.701092 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d08036-21d3-4e30-94a0-baeedd3ac721 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:50.701734 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d08036-21d3-4e30-94a0-baeedd3ac721 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:50.702017 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d08036-21d3-4e30-94a0-baeedd3ac721 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:50.702294 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d08036-21d3-4e30-94a0-baeedd3ac721 None admin] PATH_INFO: `/v3/users/fd491fe622e34c4aba04353a67b81134` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:50.711816 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674440b9-7c38-43cf-b7c8-af983ba2a1c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:50.713024 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674440b9-7c38-43cf-b7c8-af983ba2a1c3 None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:50.713511 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674440b9-7c38-43cf-b7c8-af983ba2a1c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:50.714107 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-674440b9-7c38-43cf-b7c8-af983ba2a1c3 None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:50.715346 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-674440b9-7c38-43cf-b7c8-af983ba2a1c3 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:50.716908 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7d08036-21d3-4e30-94a0-baeedd3ac721 None admin] RBAC: Authorizing `identity:get_user(user_id=fd491fe622e34c4aba04353a67b81134)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:50.717525 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-674440b9-7c38-43cf-b7c8-af983ba2a1c3 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:50.718526 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7d08036-21d3-4e30-94a0-baeedd3ac721 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:50.720778 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 139/279] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:30:50 2022] GET /identity/v3/users/fd491fe622e34c4aba04353a67b81134 => generated 293 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:50.727703 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d08036-21d3-4e30-94a0-baeedd3ac721 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:50.745277 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 141/280] 10.222.0.25 () {60 vars in 1132 bytes} [Fri Oct 28 09:30:50 2022] POST /identity/v3/users => generated 293 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:30:50.811703 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674440b9-7c38-43cf-b7c8-af983ba2a1c3 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:50.868551 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d08036-21d3-4e30-94a0-baeedd3ac721 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:50.873180 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:50.891989 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:50.922095 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deaddada-0f42-4f4e-85df-a29c91b0a3c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:50.922807 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deaddada-0f42-4f4e-85df-a29c91b0a3c1 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:50.923403 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deaddada-0f42-4f4e-85df-a29c91b0a3c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:50.923680 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deaddada-0f42-4f4e-85df-a29c91b0a3c1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:50.925007 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deaddada-0f42-4f4e-85df-a29c91b0a3c1 None admin] RBAC: Adding query filter params (role.id=8943f289c9fe4feba2695b9c90cf3d51, user.id=fd491fe622e34c4aba04353a67b81134, scope.system=all) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:50.925325 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deaddada-0f42-4f4e-85df-a29c91b0a3c1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:50.927107 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deaddada-0f42-4f4e-85df-a29c91b0a3c1 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:50.938566 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 140/281] 10.222.0.25 () {56 vars in 1308 bytes} [Fri Oct 28 09:30:50 2022] GET /identity/v3/role_assignments?user.id=fd491fe622e34c4aba04353a67b81134&scope.system=all&role.id=8943f289c9fe4feba2695b9c90cf3d51 => generated 226 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:50.955935 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-674440b9-7c38-43cf-b7c8-af983ba2a1c3 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:50.999212 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af583c8-43c6-488b-b46d-c03949d1097f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:50.999835 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af583c8-43c6-488b-b46d-c03949d1097f None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:51.000180 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af583c8-43c6-488b-b46d-c03949d1097f None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:51.000460 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af583c8-43c6-488b-b46d-c03949d1097f None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:51.014803 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8af583c8-43c6-488b-b46d-c03949d1097f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:30:51.020785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-8af583c8-43c6-488b-b46d-c03949d1097f None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 09:30:51.020785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:30:51.020785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:30:51.020785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:30:51.020785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:30:51.020785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:30:51.020785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:30:51.020785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:30:51.020785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:30:51.020785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:30:51.020785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:30:51.020785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:30:51.020785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:30:51.020785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:30:51.020785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:30:51.020785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:30:51.021743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:30:51.021743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:30:51.021743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:30:51.021743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:30:51.021743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:30:51.021743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:30:51.021743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:30:51.021743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:30:51.021743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:30:51.021743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:30:51.021743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:30:51.021743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:30:51.021743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:30:51.021743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:30:51.021743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:30:51.021743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:30:51.022993 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:30:51.022993 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:30:51.022993 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:30:51.022993 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:30:51.022993 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:30:51.022993 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:30:51.022993 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:30:51.022993 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:30:51.022993 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:30:51.022993 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:30:51.022993 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 09:30:51.022993 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:30:51.022993 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 142/282] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:30:50 2022] GET /identity/v3/roles/reader => generated 84 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:30:51.028300 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deaddada-0f42-4f4e-85df-a29c91b0a3c1 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:51.147306 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deaddada-0f42-4f4e-85df-a29c91b0a3c1 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:51.185390 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f862d2-4c33-42a1-8cda-0b56fb0ed980 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:51.186009 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f862d2-4c33-42a1-8cda-0b56fb0ed980 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:51.186354 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f862d2-4c33-42a1-8cda-0b56fb0ed980 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:51.186660 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f862d2-4c33-42a1-8cda-0b56fb0ed980 None admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:51.187337 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f862d2-4c33-42a1-8cda-0b56fb0ed980 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:51.187643 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f862d2-4c33-42a1-8cda-0b56fb0ed980 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:51.189101 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03f862d2-4c33-42a1-8cda-0b56fb0ed980 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:51.199102 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 141/283] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Oct 28 09:30:51 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:51.204780 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af583c8-43c6-488b-b46d-c03949d1097f None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:51.322976 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af583c8-43c6-488b-b46d-c03949d1097f None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:51.325394 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:51.355955 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da25066-8ba1-48ff-a6ab-28168f0789e9 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:51.356262 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da25066-8ba1-48ff-a6ab-28168f0789e9 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:51.356570 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da25066-8ba1-48ff-a6ab-28168f0789e9 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:51.358073 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 142/284] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:30:51.363202 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918fa798-68f1-4495-8de9-f8b17df58aff None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:51.363498 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918fa798-68f1-4495-8de9-f8b17df58aff None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:51.363776 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918fa798-68f1-4495-8de9-f8b17df58aff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:51.374762 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1043ec4-afa6-4df4-a2cc-7550f95cfbc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:51.375412 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1043ec4-afa6-4df4-a2cc-7550f95cfbc9 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:51.375696 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1043ec4-afa6-4df4-a2cc-7550f95cfbc9 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:51.375973 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1043ec4-afa6-4df4-a2cc-7550f95cfbc9 None admin] PATH_INFO: `/v3/users/ac5c183a78ce4f15bb0191916824c968` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:51.378854 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1043ec4-afa6-4df4-a2cc-7550f95cfbc9 None admin] RBAC: Authorizing `identity:get_user(user_id=ac5c183a78ce4f15bb0191916824c968)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:51.380021 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1043ec4-afa6-4df4-a2cc-7550f95cfbc9 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:51.381923 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 143/285] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:30:51 2022] GET /identity/v3/users/ac5c183a78ce4f15bb0191916824c968 => generated 297 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:51.386318 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-918fa798-68f1-4495-8de9-f8b17df58aff None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:51.387886 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1043ec4-afa6-4df4-a2cc-7550f95cfbc9 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:51.388392 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-918fa798-68f1-4495-8de9-f8b17df58aff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:51.434197 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:51.477305 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 143/286] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 09:30:51.531635 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1043ec4-afa6-4df4-a2cc-7550f95cfbc9 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:51.591538 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-577dac90-d977-4b00-97c6-6c07a8777100 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:51.592185 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577dac90-d977-4b00-97c6-6c07a8777100 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:51.592464 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577dac90-d977-4b00-97c6-6c07a8777100 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:51.592741 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577dac90-d977-4b00-97c6-6c07a8777100 None admin] PATH_INFO: `/v3/projects/5960af42464f4e83b60ec52264d04b70` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:51.596849 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-577dac90-d977-4b00-97c6-6c07a8777100 None admin] RBAC: Authorizing `identity:get_project(project_id=5960af42464f4e83b60ec52264d04b70)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:51.598316 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-577dac90-d977-4b00-97c6-6c07a8777100 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:51.600382 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 144/287] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:30:51 2022] GET /identity/v3/projects/5960af42464f4e83b60ec52264d04b70 => generated 302 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:30:51.605794 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918fa798-68f1-4495-8de9-f8b17df58aff None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:51.741128 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918fa798-68f1-4495-8de9-f8b17df58aff None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:51.780207 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000da5b9-f689-4d09-ad29-453bbcfe4a7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:51.780876 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000da5b9-f689-4d09-ad29-453bbcfe4a7e None admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:51.781155 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000da5b9-f689-4d09-ad29-453bbcfe4a7e None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:51.781431 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000da5b9-f689-4d09-ad29-453bbcfe4a7e None admin] PATH_INFO: `/v3/projects/5960af42464f4e83b60ec52264d04b70/users/ac5c183a78ce4f15bb0191916824c968/roles/12f06a0e43d345aba46a51c26c012ce3` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:51.787701 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-000da5b9-f689-4d09-ad29-453bbcfe4a7e None admin] RBAC: Authorizing `identity:create_grant(project_id=5960af42464f4e83b60ec52264d04b70, user_id=ac5c183a78ce4f15bb0191916824c968, role_id=12f06a0e43d345aba46a51c26c012ce3)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:51.789211 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-000da5b9-f689-4d09-ad29-453bbcfe4a7e None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:51.806414 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 144/288] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Oct 28 09:30:51 2022] PUT /identity/v3/projects/5960af42464f4e83b60ec52264d04b70/users/ac5c183a78ce4f15bb0191916824c968/roles/12f06a0e43d345aba46a51c26c012ce3 => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:30:51.840314 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-577dac90-d977-4b00-97c6-6c07a8777100 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:51.902475 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:51.980912 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-577dac90-d977-4b00-97c6-6c07a8777100 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:52.017244 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fdd199a-498b-48bb-9d56-f2128f248b31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:52.017889 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdd199a-498b-48bb-9d56-f2128f248b31 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:52.018168 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdd199a-498b-48bb-9d56-f2128f248b31 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:52.018470 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdd199a-498b-48bb-9d56-f2128f248b31 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:52.029811 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fdd199a-498b-48bb-9d56-f2128f248b31 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:30:52.035725 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-5fdd199a-498b-48bb-9d56-f2128f248b31 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 09:30:52.035725 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:30:52.035725 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:30:52.035725 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:30:52.035725 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:30:52.035725 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:30:52.035725 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:30:52.035725 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:30:52.035725 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:30:52.035725 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:30:52.035725 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:30:52.035725 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:30:52.035725 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:30:52.035725 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:30:52.035725 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:30:52.035725 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:30:52.036675 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:30:52.036675 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:30:52.036675 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:30:52.036675 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:30:52.036675 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:30:52.036675 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:30:52.036675 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:30:52.036675 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:30:52.036675 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:30:52.036675 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:30:52.036675 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:30:52.036675 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:30:52.036675 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:30:52.036675 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:30:52.036675 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:30:52.036675 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:30:52.037533 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:30:52.037533 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:30:52.037533 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:30:52.037533 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:30:52.037533 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:30:52.037533 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:30:52.037533 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:30:52.037533 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:30:52.037533 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:30:52.037533 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:30:52.037533 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 09:30:52.037533 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:30:52.037533 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 145/289] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:30:51 2022] GET /identity/v3/roles/member => generated 84 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:30:52.045283 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000da5b9-f689-4d09-ad29-453bbcfe4a7e None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:52.113137 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:52.199789 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000da5b9-f689-4d09-ad29-453bbcfe4a7e None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:52.236441 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a7a79e7-f071-46a6-8847-ded4cdd32db3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:52.237426 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7a79e7-f071-46a6-8847-ded4cdd32db3 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:52.237908 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7a79e7-f071-46a6-8847-ded4cdd32db3 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:52.238445 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7a79e7-f071-46a6-8847-ded4cdd32db3 None admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:52.239324 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a7a79e7-f071-46a6-8847-ded4cdd32db3 None admin] RBAC: Adding query filter params (name=member) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:52.239834 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a7a79e7-f071-46a6-8847-ded4cdd32db3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:52.241497 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a7a79e7-f071-46a6-8847-ded4cdd32db3 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:52.252619 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 145/290] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Oct 28 09:30:52 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:52.260985 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fdd199a-498b-48bb-9d56-f2128f248b31 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:52.370320 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:52.374570 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fdd199a-498b-48bb-9d56-f2128f248b31 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:52.417772 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614d6b2b-8384-495e-bc19-ca8807ae6c87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:52.418773 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614d6b2b-8384-495e-bc19-ca8807ae6c87 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:52.419277 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614d6b2b-8384-495e-bc19-ca8807ae6c87 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:52.419746 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614d6b2b-8384-495e-bc19-ca8807ae6c87 None admin] PATH_INFO: `/v3/users/1ac90379b2454e689b29dc7e52c529a7` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:52.421157 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-614d6b2b-8384-495e-bc19-ca8807ae6c87 None admin] RBAC: Authorizing `identity:get_user(user_id=1ac90379b2454e689b29dc7e52c529a7)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:52.422880 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-614d6b2b-8384-495e-bc19-ca8807ae6c87 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:52.424877 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 146/291] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:30:52 2022] GET /identity/v3/users/1ac90379b2454e689b29dc7e52c529a7 => generated 297 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:52.431201 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a7a79e7-f071-46a6-8847-ded4cdd32db3 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:52.558983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:52.568440 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a7a79e7-f071-46a6-8847-ded4cdd32db3 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:52.612421 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf9ae1e-bfa8-4d9e-8d0c-03288b2a7094 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:52.613047 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf9ae1e-bfa8-4d9e-8d0c-03288b2a7094 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:52.613332 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf9ae1e-bfa8-4d9e-8d0c-03288b2a7094 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:52.613611 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf9ae1e-bfa8-4d9e-8d0c-03288b2a7094 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:52.702002 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf9ae1e-bfa8-4d9e-8d0c-03288b2a7094 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:52.703421 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf9ae1e-bfa8-4d9e-8d0c-03288b2a7094 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:52.823666 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:52.835295 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 146/292] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:30:52 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 405 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 09:30:52.841276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614d6b2b-8384-495e-bc19-ca8807ae6c87 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:52.946031 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614d6b2b-8384-495e-bc19-ca8807ae6c87 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:52.979763 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-481b95b6-cd96-4174-9bc8-93638c264cc6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:52.980315 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481b95b6-cd96-4174-9bc8-93638c264cc6 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:52.980592 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481b95b6-cd96-4174-9bc8-93638c264cc6 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:52.980862 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481b95b6-cd96-4174-9bc8-93638c264cc6 None admin] PATH_INFO: `/v3/projects/5960af42464f4e83b60ec52264d04b70` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:52.981979 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-481b95b6-cd96-4174-9bc8-93638c264cc6 None admin] RBAC: Authorizing `identity:get_project(project_id=5960af42464f4e83b60ec52264d04b70)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:52.983438 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-481b95b6-cd96-4174-9bc8-93638c264cc6 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:52.985419 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 147/293] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:30:52 2022] GET /identity/v3/projects/5960af42464f4e83b60ec52264d04b70 => generated 302 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:52.991266 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf9ae1e-bfa8-4d9e-8d0c-03288b2a7094 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:53.108126 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf9ae1e-bfa8-4d9e-8d0c-03288b2a7094 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:53.144082 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e2f4f2d-d923-4217-a812-4dca4c32fe53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:53.144695 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2f4f2d-d923-4217-a812-4dca4c32fe53 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:53.144972 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2f4f2d-d923-4217-a812-4dca4c32fe53 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:53.145242 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2f4f2d-d923-4217-a812-4dca4c32fe53 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:53.146017 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e2f4f2d-d923-4217-a812-4dca4c32fe53 None admin] RBAC: Adding query filter params (role.id=8943f289c9fe4feba2695b9c90cf3d51, scope.project.id=5960af42464f4e83b60ec52264d04b70, user.id=1ac90379b2454e689b29dc7e52c529a7) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:53.146332 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e2f4f2d-d923-4217-a812-4dca4c32fe53 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:53.147785 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e2f4f2d-d923-4217-a812-4dca4c32fe53 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:53.154096 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 147/294] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:30:52 2022] GET /identity/v3/role_assignments?user.id=1ac90379b2454e689b29dc7e52c529a7&scope.project.id=5960af42464f4e83b60ec52264d04b70&role.id=8943f289c9fe4feba2695b9c90cf3d51 => generated 608 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:53.563537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2527d4-1aa6-4e71-beb9-3b1f2e3b4b3c None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:53.563898 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2527d4-1aa6-4e71-beb9-3b1f2e3b4b3c None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:53.564119 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2527d4-1aa6-4e71-beb9-3b1f2e3b4b3c None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:53.565460 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 148/295] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:30:53.572905 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673afeef-3574-4873-90dd-74afc4c42194 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:53.572905 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673afeef-3574-4873-90dd-74afc4c42194 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:53.572905 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-673afeef-3574-4873-90dd-74afc4c42194 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:53.595038 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-673afeef-3574-4873-90dd-74afc4c42194 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:53.596374 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-673afeef-3574-4873-90dd-74afc4c42194 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:53.667322 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:53.685193 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 148/296] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:30:53.759873 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da77b5fa-1935-4a8c-b0e6-50da14328ef9 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:53.760201 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da77b5fa-1935-4a8c-b0e6-50da14328ef9 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:53.760449 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da77b5fa-1935-4a8c-b0e6-50da14328ef9 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:53.762073 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 149/297] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:30:53.767545 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239b7534-6ecd-41e3-80e9-1f407e5c8b80 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:53.767829 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239b7534-6ecd-41e3-80e9-1f407e5c8b80 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:53.768103 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-239b7534-6ecd-41e3-80e9-1f407e5c8b80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:53.791075 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-239b7534-6ecd-41e3-80e9-1f407e5c8b80 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:53.792266 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-239b7534-6ecd-41e3-80e9-1f407e5c8b80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:53.870588 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 149/298] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:30:53.969040 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da77b5fa-1935-4a8c-b0e6-50da14328ef9 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:53.972245 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d775f292-6e76-49be-a662-337194f785f5 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:53.973699 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d775f292-6e76-49be-a662-337194f785f5 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:53.974189 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d775f292-6e76-49be-a662-337194f785f5 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:53.975775 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 150/299] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:53 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 09:30:53.981663 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0efd3d97-c2de-45e2-ad8e-838fbe8c66d3 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:53.982323 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0efd3d97-c2de-45e2-ad8e-838fbe8c66d3 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:53.982715 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0efd3d97-c2de-45e2-ad8e-838fbe8c66d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:54.002220 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-0efd3d97-c2de-45e2-ad8e-838fbe8c66d3 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:54.003626 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-0efd3d97-c2de-45e2-ad8e-838fbe8c66d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:54.006844 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:54.085016 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 151/300] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 09:30:54.093825 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da77b5fa-1935-4a8c-b0e6-50da14328ef9 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:54.128676 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861f7d6f-958b-413a-b166-a34478e504ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:54.129215 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861f7d6f-958b-413a-b166-a34478e504ec None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:54.129499 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861f7d6f-958b-413a-b166-a34478e504ec None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:54.129781 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861f7d6f-958b-413a-b166-a34478e504ec None admin] PATH_INFO: `/v3/roles/member` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:54.141091 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-861f7d6f-958b-413a-b166-a34478e504ec None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:30:54.145993 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-861f7d6f-958b-413a-b166-a34478e504ec None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 09:30:54.145993 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:30:54.145993 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:30:54.145993 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:30:54.145993 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:30:54.145993 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:30:54.145993 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:30:54.145993 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:30:54.145993 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:30:54.145993 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:30:54.145993 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:30:54.145993 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:30:54.145993 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:30:54.145993 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:30:54.145993 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:30:54.145993 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:30:54.146903 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:30:54.146903 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:30:54.146903 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:30:54.146903 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:30:54.146903 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:30:54.146903 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:30:54.146903 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:30:54.146903 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:30:54.146903 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:30:54.146903 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:30:54.146903 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:30:54.146903 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:30:54.146903 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:30:54.146903 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:30:54.146903 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:30:54.146903 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:30:54.147682 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:30:54.147682 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:30:54.147682 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:30:54.147682 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:30:54.147682 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:30:54.147682 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:30:54.147682 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:30:54.147682 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:30:54.147682 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:30:54.147682 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:30:54.147682 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 09:30:54.147682 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:30:54.147682 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 150/301] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:30:53 2022] GET /identity/v3/roles/member => generated 84 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:30:54.154327 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0efd3d97-c2de-45e2-ad8e-838fbe8c66d3 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:54.166649 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861f7d6f-958b-413a-b166-a34478e504ec None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:54.213679 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:54.282687 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0efd3d97-c2de-45e2-ad8e-838fbe8c66d3 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:54.287980 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:54.296988 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861f7d6f-958b-413a-b166-a34478e504ec None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:54.326225 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81afc05-e2ee-4931-b1d7-9b67e3a98b48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:54.327213 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81afc05-e2ee-4931-b1d7-9b67e3a98b48 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:54.327585 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81afc05-e2ee-4931-b1d7-9b67e3a98b48 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:54.328281 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81afc05-e2ee-4931-b1d7-9b67e3a98b48 None admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:54.329573 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b81afc05-e2ee-4931-b1d7-9b67e3a98b48 None admin] RBAC: Adding query filter params (name=member) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:54.329973 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b81afc05-e2ee-4931-b1d7-9b67e3a98b48 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:54.331956 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b81afc05-e2ee-4931-b1d7-9b67e3a98b48 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:54.340937 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9995281-84ed-4d1b-8c1a-73fbd341ab6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:54.341779 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9995281-84ed-4d1b-8c1a-73fbd341ab6a None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:54.342223 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9995281-84ed-4d1b-8c1a-73fbd341ab6a None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:54.347029 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9995281-84ed-4d1b-8c1a-73fbd341ab6a None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:54.347154 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 152/302] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Oct 28 09:30:54 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:54.349941 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81afc05-e2ee-4931-b1d7-9b67e3a98b48 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:54.359871 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9995281-84ed-4d1b-8c1a-73fbd341ab6a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:30:54.365099 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-c9995281-84ed-4d1b-8c1a-73fbd341ab6a None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 09:30:54.365099 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:30:54.365099 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:30:54.365099 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:30:54.365099 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:30:54.365099 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:30:54.365099 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:30:54.365099 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:30:54.365099 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:30:54.365099 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:30:54.365099 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:30:54.365099 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:30:54.365099 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:30:54.365099 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:30:54.365099 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:30:54.365099 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:30:54.366276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:30:54.366276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:30:54.366276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:30:54.366276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:30:54.366276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:30:54.366276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:30:54.366276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:30:54.366276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:30:54.366276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:30:54.366276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:30:54.366276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:30:54.366276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:30:54.366276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:30:54.366276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:30:54.366276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:30:54.366276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:30:54.367646 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:30:54.367646 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:30:54.367646 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:30:54.367646 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:30:54.367646 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:30:54.367646 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:30:54.367646 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:30:54.367646 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:30:54.367646 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:30:54.367646 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:30:54.367646 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 09:30:54.367646 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:30:54.371286 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 151/303] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:30:54 2022] GET /identity/v3/roles/reader => generated 84 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:30:54.372169 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9995281-84ed-4d1b-8c1a-73fbd341ab6a None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:54.477143 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81afc05-e2ee-4931-b1d7-9b67e3a98b48 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:54.486298 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9995281-84ed-4d1b-8c1a-73fbd341ab6a None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:54.489503 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:54.521759 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e15efa4-aa50-48be-a135-0f456b55b5ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:54.522404 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e15efa4-aa50-48be-a135-0f456b55b5ad None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:54.522708 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e15efa4-aa50-48be-a135-0f456b55b5ad None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:54.522981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e15efa4-aa50-48be-a135-0f456b55b5ad None admin] PATH_INFO: `/v3/users/fd491fe622e34c4aba04353a67b81134` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:54.524266 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e15efa4-aa50-48be-a135-0f456b55b5ad None admin] RBAC: Authorizing `identity:get_user(user_id=fd491fe622e34c4aba04353a67b81134)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:54.525734 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e15efa4-aa50-48be-a135-0f456b55b5ad None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:54.527399 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 153/304] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:30:54 2022] GET /identity/v3/users/fd491fe622e34c4aba04353a67b81134 => generated 293 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:54.527399 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e15efa4-aa50-48be-a135-0f456b55b5ad None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:54.535742 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35fd51f9-4844-488f-baf2-77d41bd5260d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:54.536382 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35fd51f9-4844-488f-baf2-77d41bd5260d None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:54.536661 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35fd51f9-4844-488f-baf2-77d41bd5260d None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:54.536931 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35fd51f9-4844-488f-baf2-77d41bd5260d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:54.538061 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35fd51f9-4844-488f-baf2-77d41bd5260d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:54.540778 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35fd51f9-4844-488f-baf2-77d41bd5260d None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:54.543437 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 152/305] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:30:54 2022] GET /identity/v3/domains/default => generated 199 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:54.544305 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35fd51f9-4844-488f-baf2-77d41bd5260d None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:54.584365 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:54.675528 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e15efa4-aa50-48be-a135-0f456b55b5ad None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:54.685167 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35fd51f9-4844-488f-baf2-77d41bd5260d None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:54.718024 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b56e9ba-47be-45ed-9c5a-dba74ebda8b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:54.718675 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b56e9ba-47be-45ed-9c5a-dba74ebda8b4 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:54.718954 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b56e9ba-47be-45ed-9c5a-dba74ebda8b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:54.719236 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b56e9ba-47be-45ed-9c5a-dba74ebda8b4 None admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:54.719884 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b56e9ba-47be-45ed-9c5a-dba74ebda8b4 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:54.720191 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b56e9ba-47be-45ed-9c5a-dba74ebda8b4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:54.721551 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b56e9ba-47be-45ed-9c5a-dba74ebda8b4 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:54.722979 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560f1d4c-2a7f-48ff-b42f-a2664263fd37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:54.723641 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560f1d4c-2a7f-48ff-b42f-a2664263fd37 None admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:54.723930 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560f1d4c-2a7f-48ff-b42f-a2664263fd37 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:54.724206 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560f1d4c-2a7f-48ff-b42f-a2664263fd37 None admin] PATH_INFO: `/v3/system/users/fd491fe622e34c4aba04353a67b81134/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:54.725639 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-560f1d4c-2a7f-48ff-b42f-a2664263fd37 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=fd491fe622e34c4aba04353a67b81134, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:54.727070 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-560f1d4c-2a7f-48ff-b42f-a2664263fd37 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:54.732487 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 154/306] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Oct 28 09:30:54 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:54.734375 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b56e9ba-47be-45ed-9c5a-dba74ebda8b4 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:54.737891 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 153/307] 10.222.0.25 () {58 vars in 1334 bytes} [Fri Oct 28 09:30:54 2022] PUT /identity/v3/system/users/fd491fe622e34c4aba04353a67b81134/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:30:54.744624 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f200f729-d520-4d40-bc28-6a1b7f45151b None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:54.745349 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f200f729-d520-4d40-bc28-6a1b7f45151b None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:54.745712 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f200f729-d520-4d40-bc28-6a1b7f45151b None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:54.753141 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 154/308] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:54 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:30:54.753141 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f200f729-d520-4d40-bc28-6a1b7f45151b None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:54.790450 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:54.868032 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b56e9ba-47be-45ed-9c5a-dba74ebda8b4 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:54.878564 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:54.898820 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f200f729-d520-4d40-bc28-6a1b7f45151b None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:54.925133 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02eaeba-eb7d-456d-a778-4ea63920c205 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:54.925750 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02eaeba-eb7d-456d-a778-4ea63920c205 None admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:54.926520 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02eaeba-eb7d-456d-a778-4ea63920c205 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:54.926520 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02eaeba-eb7d-456d-a778-4ea63920c205 None admin] PATH_INFO: `/v3/groups` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:54.926918 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c02eaeba-eb7d-456d-a778-4ea63920c205 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:54.928963 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c02eaeba-eb7d-456d-a778-4ea63920c205 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:54.949632 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 155/309] 10.222.0.25 () {60 vars in 1134 bytes} [Fri Oct 28 09:30:54 2022] POST /identity/v3/groups => generated 229 bytes in 217 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:30:54.950884 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4456e779-2ae5-44cb-a9a0-121e31d989ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:54.951958 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4456e779-2ae5-44cb-a9a0-121e31d989ad None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:54.952233 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4456e779-2ae5-44cb-a9a0-121e31d989ad None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:54.952597 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4456e779-2ae5-44cb-a9a0-121e31d989ad None admin] PATH_INFO: `/v3/users/c8b321012884465881f5e564d2d76d7b` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:54.959115 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782700e9-17ad-43d3-9c6f-753e41f2fe94 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:54.959486 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782700e9-17ad-43d3-9c6f-753e41f2fe94 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:54.959679 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782700e9-17ad-43d3-9c6f-753e41f2fe94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:54.974025 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4456e779-2ae5-44cb-a9a0-121e31d989ad None admin] RBAC: Authorizing `identity:get_user(user_id=c8b321012884465881f5e564d2d76d7b)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:54.976130 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4456e779-2ae5-44cb-a9a0-121e31d989ad None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:54.978447 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 155/310] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:30:54 2022] GET /identity/v3/users/c8b321012884465881f5e564d2d76d7b => generated 293 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:54.986229 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4456e779-2ae5-44cb-a9a0-121e31d989ad None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:54.989304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-782700e9-17ad-43d3-9c6f-753e41f2fe94 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:54.991118 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-782700e9-17ad-43d3-9c6f-753e41f2fe94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:55.123122 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 156/311] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:30:55.149246 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4456e779-2ae5-44cb-a9a0-121e31d989ad None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:55.193500 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c18b4a7-b9bf-4495-b1d5-cb17704b62b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:55.194507 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c18b4a7-b9bf-4495-b1d5-cb17704b62b8 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:55.195010 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c18b4a7-b9bf-4495-b1d5-cb17704b62b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:55.196015 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c18b4a7-b9bf-4495-b1d5-cb17704b62b8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:55.196957 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c18b4a7-b9bf-4495-b1d5-cb17704b62b8 None admin] RBAC: Adding query filter params (role.id=12f06a0e43d345aba46a51c26c012ce3, user.id=c8b321012884465881f5e564d2d76d7b, scope.system=all) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:55.197460 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c18b4a7-b9bf-4495-b1d5-cb17704b62b8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:55.199116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c18b4a7-b9bf-4495-b1d5-cb17704b62b8 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:55.206448 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 156/312] 10.222.0.25 () {56 vars in 1308 bytes} [Fri Oct 28 09:30:54 2022] GET /identity/v3/role_assignments?user.id=c8b321012884465881f5e564d2d76d7b&scope.system=all&role.id=12f06a0e43d345aba46a51c26c012ce3 => generated 226 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:55.399174 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-782700e9-17ad-43d3-9c6f-753e41f2fe94 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:55.430595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:55.526943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-782700e9-17ad-43d3-9c6f-753e41f2fe94 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:55.569378 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86526b49-e99c-4265-a8a2-44199229a917 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:55.570519 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86526b49-e99c-4265-a8a2-44199229a917 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:55.571268 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86526b49-e99c-4265-a8a2-44199229a917 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:55.572000 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86526b49-e99c-4265-a8a2-44199229a917 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:55.587287 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86526b49-e99c-4265-a8a2-44199229a917 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:30:55.595837 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-86526b49-e99c-4265-a8a2-44199229a917 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 09:30:55.595837 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:30:55.595837 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:30:55.595837 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:30:55.595837 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:30:55.595837 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:30:55.595837 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:30:55.595837 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:30:55.595837 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:30:55.595837 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:30:55.595837 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:30:55.595837 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:30:55.595837 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:30:55.595837 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:30:55.595837 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:30:55.595837 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:30:55.597100 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:30:55.597100 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:30:55.597100 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:30:55.597100 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:30:55.597100 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:30:55.597100 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:30:55.597100 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:30:55.597100 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:30:55.597100 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:30:55.597100 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:30:55.597100 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:30:55.597100 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:30:55.597100 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:30:55.597100 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:30:55.597100 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:30:55.597100 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:30:55.597874 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:30:55.597874 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:30:55.597874 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:30:55.597874 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:30:55.597874 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:30:55.597874 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:30:55.597874 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:30:55.597874 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:30:55.597874 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:30:55.597874 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:30:55.597874 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 09:30:55.597874 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:30:55.600824 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 157/313] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:30:55 2022] GET /identity/v3/roles/reader => generated 84 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:30:55.608985 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c18b4a7-b9bf-4495-b1d5-cb17704b62b8 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:55.625512 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:55.723543 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c18b4a7-b9bf-4495-b1d5-cb17704b62b8 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:55.760192 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd99a44-92ef-48d7-968e-3f3ac62e68fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:55.761227 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd99a44-92ef-48d7-968e-3f3ac62e68fa None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:55.761738 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd99a44-92ef-48d7-968e-3f3ac62e68fa None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:55.762210 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd99a44-92ef-48d7-968e-3f3ac62e68fa None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:55.763094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dd99a44-92ef-48d7-968e-3f3ac62e68fa None admin] RBAC: Adding query filter params (name=reader) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:55.763611 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dd99a44-92ef-48d7-968e-3f3ac62e68fa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:55.765275 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dd99a44-92ef-48d7-968e-3f3ac62e68fa None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:55.775519 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 157/314] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Oct 28 09:30:55 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:55.781550 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86526b49-e99c-4265-a8a2-44199229a917 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:55.895873 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:55.912970 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86526b49-e99c-4265-a8a2-44199229a917 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:55.962033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-043e5bb0-0f49-46f1-838c-8175326e8584 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:55.963702 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043e5bb0-0f49-46f1-838c-8175326e8584 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:55.964191 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043e5bb0-0f49-46f1-838c-8175326e8584 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:55.964667 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043e5bb0-0f49-46f1-838c-8175326e8584 None admin] PATH_INFO: `/v3/users/ac5c183a78ce4f15bb0191916824c968` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:55.967105 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-043e5bb0-0f49-46f1-838c-8175326e8584 None admin] RBAC: Authorizing `identity:get_user(user_id=ac5c183a78ce4f15bb0191916824c968)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:55.969412 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-043e5bb0-0f49-46f1-838c-8175326e8584 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:55.971855 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 158/315] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:30:55 2022] GET /identity/v3/users/ac5c183a78ce4f15bb0191916824c968 => generated 297 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:55.981322 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd99a44-92ef-48d7-968e-3f3ac62e68fa None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:56.084596 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:56.126526 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd99a44-92ef-48d7-968e-3f3ac62e68fa None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:56.174458 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9915da8-7929-43ca-8c76-34d9f303fc45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:56.175445 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9915da8-7929-43ca-8c76-34d9f303fc45 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:56.175823 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9915da8-7929-43ca-8c76-34d9f303fc45 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:56.176107 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9915da8-7929-43ca-8c76-34d9f303fc45 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:56.266808 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9915da8-7929-43ca-8c76-34d9f303fc45 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:56.268188 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9915da8-7929-43ca-8c76-34d9f303fc45 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:56.340774 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:56.373452 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 158/316] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:30:55 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 393 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 09:30:56.379176 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-043e5bb0-0f49-46f1-838c-8175326e8584 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:56.526726 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-043e5bb0-0f49-46f1-838c-8175326e8584 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:56.586694 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e30b5b7-8f68-437b-81f4-390de9fab5a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:56.587738 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e30b5b7-8f68-437b-81f4-390de9fab5a1 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:56.588336 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e30b5b7-8f68-437b-81f4-390de9fab5a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:56.588853 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e30b5b7-8f68-437b-81f4-390de9fab5a1 None admin] PATH_INFO: `/v3/projects/5960af42464f4e83b60ec52264d04b70` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:56.590352 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e30b5b7-8f68-437b-81f4-390de9fab5a1 None admin] RBAC: Authorizing `identity:get_project(project_id=5960af42464f4e83b60ec52264d04b70)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:56.592763 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e30b5b7-8f68-437b-81f4-390de9fab5a1 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:56.596351 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 159/317] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:30:56 2022] GET /identity/v3/projects/5960af42464f4e83b60ec52264d04b70 => generated 302 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:56.603554 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9915da8-7929-43ca-8c76-34d9f303fc45 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:56.708503 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9915da8-7929-43ca-8c76-34d9f303fc45 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:56.745572 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f21439a7-c220-4be2-84a6-609f688fcfb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:56.746194 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21439a7-c220-4be2-84a6-609f688fcfb4 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:56.746504 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21439a7-c220-4be2-84a6-609f688fcfb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:56.746828 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21439a7-c220-4be2-84a6-609f688fcfb4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:56.747576 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f21439a7-c220-4be2-84a6-609f688fcfb4 None admin] RBAC: Adding query filter params (role.id=12f06a0e43d345aba46a51c26c012ce3, scope.project.id=5960af42464f4e83b60ec52264d04b70, user.id=ac5c183a78ce4f15bb0191916824c968) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:56.747887 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f21439a7-c220-4be2-84a6-609f688fcfb4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:56.749378 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f21439a7-c220-4be2-84a6-609f688fcfb4 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:56.756135 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 159/318] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:30:56 2022] GET /identity/v3/role_assignments?user.id=ac5c183a78ce4f15bb0191916824c968&scope.project.id=5960af42464f4e83b60ec52264d04b70&role.id=12f06a0e43d345aba46a51c26c012ce3 => generated 608 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:57.012523 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9721ea5-fad5-44e5-a733-860106fc8155 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:57.013064 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9721ea5-fad5-44e5-a733-860106fc8155 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:57.013064 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9721ea5-fad5-44e5-a733-860106fc8155 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:57.014659 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 160/319] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:30:57.023186 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b727f1-9931-4471-bfd2-d8b018b65221 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:57.024153 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b727f1-9931-4471-bfd2-d8b018b65221 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:57.024440 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b727f1-9931-4471-bfd2-d8b018b65221 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:57.047075 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-e1b727f1-9931-4471-bfd2-d8b018b65221 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:57.048590 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e1b727f1-9931-4471-bfd2-d8b018b65221 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:57.087406 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:57.139320 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 160/320] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:30:57.420146 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9721ea5-fad5-44e5-a733-860106fc8155 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:57.456918 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:57.466042 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3cb572a-ec00-452f-b8e5-db0ff8af32bc None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:57.466371 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3cb572a-ec00-452f-b8e5-db0ff8af32bc None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:57.467016 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3cb572a-ec00-452f-b8e5-db0ff8af32bc None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:57.469413 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 161/321] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 09:30:57.475079 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b0c669-b42d-4bb1-b68e-e82cf4f2790b None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:57.475481 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b0c669-b42d-4bb1-b68e-e82cf4f2790b None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:57.475954 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b0c669-b42d-4bb1-b68e-e82cf4f2790b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:57.494839 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-54b0c669-b42d-4bb1-b68e-e82cf4f2790b None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:57.496835 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-54b0c669-b42d-4bb1-b68e-e82cf4f2790b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:57.555335 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9721ea5-fad5-44e5-a733-860106fc8155 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:57.580051 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 162/322] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 09:30:57.582343 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76788159-ba96-4de4-8508-348ee3b8d405 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:57.582343 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76788159-ba96-4de4-8508-348ee3b8d405 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:57.582343 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76788159-ba96-4de4-8508-348ee3b8d405 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:57.583284 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 163/323] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:30:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:30:57.590425 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052aa784-16fd-46ef-85ba-66ea3b758c1c None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:57.590748 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052aa784-16fd-46ef-85ba-66ea3b758c1c None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:57.591033 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-052aa784-16fd-46ef-85ba-66ea3b758c1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:57.604747 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc16a112-f418-432a-a650-c553407c44c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:57.605396 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc16a112-f418-432a-a650-c553407c44c5 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:57.605673 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc16a112-f418-432a-a650-c553407c44c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:57.605949 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc16a112-f418-432a-a650-c553407c44c5 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:57.614359 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-052aa784-16fd-46ef-85ba-66ea3b758c1c None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:30:57.616241 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-052aa784-16fd-46ef-85ba-66ea3b758c1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:30:57.616990 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc16a112-f418-432a-a650-c553407c44c5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:30:57.621971 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-cc16a112-f418-432a-a650-c553407c44c5 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 09:30:57.621971 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:30:57.621971 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:30:57.621971 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:30:57.621971 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:30:57.621971 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:30:57.621971 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:30:57.621971 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:30:57.621971 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:30:57.621971 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:30:57.621971 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:30:57.621971 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:30:57.621971 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:30:57.621971 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:30:57.621971 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:30:57.621971 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:30:57.622999 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:30:57.622999 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:30:57.622999 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:30:57.622999 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:30:57.622999 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:30:57.622999 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:30:57.622999 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:30:57.622999 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:30:57.622999 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:30:57.622999 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:30:57.622999 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:30:57.622999 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:30:57.622999 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:30:57.622999 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:30:57.622999 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:30:57.622999 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:30:57.628547 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:30:57.628547 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:30:57.628547 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:30:57.628547 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:30:57.628547 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:30:57.628547 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:30:57.628547 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:30:57.628547 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:30:57.628547 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:30:57.628547 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:30:57.628547 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 09:30:57.628547 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:30:57.628547 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 161/324] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:30:57 2022] GET /identity/v3/roles/member => generated 84 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:30:57.630565 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc16a112-f418-432a-a650-c553407c44c5 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:57.715557 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 164/325] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:30:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 09:30:57.736520 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:57.761560 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc16a112-f418-432a-a650-c553407c44c5 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:57.808372 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5b0c62-8487-415b-bf34-142e9767906b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:57.808994 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5b0c62-8487-415b-bf34-142e9767906b None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:57.809273 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5b0c62-8487-415b-bf34-142e9767906b None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:57.809548 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5b0c62-8487-415b-bf34-142e9767906b None admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:57.810222 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b5b0c62-8487-415b-bf34-142e9767906b None admin] RBAC: Adding query filter params (name=member) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:57.810558 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b5b0c62-8487-415b-bf34-142e9767906b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:57.812080 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b5b0c62-8487-415b-bf34-142e9767906b None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:57.830426 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 162/326] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Oct 28 09:30:57 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:57.834432 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052aa784-16fd-46ef-85ba-66ea3b758c1c None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:57.849290 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5b0c62-8487-415b-bf34-142e9767906b None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:57.881599 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:57.973365 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-052aa784-16fd-46ef-85ba-66ea3b758c1c None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:57.987986 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5b0c62-8487-415b-bf34-142e9767906b None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:58.012296 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2cc4721-67b4-484f-920e-a341708eac6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:58.012927 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2cc4721-67b4-484f-920e-a341708eac6a None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:58.013204 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2cc4721-67b4-484f-920e-a341708eac6a None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:58.013473 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2cc4721-67b4-484f-920e-a341708eac6a None admin] PATH_INFO: `/v3/users/fd491fe622e34c4aba04353a67b81134` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:58.014632 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2cc4721-67b4-484f-920e-a341708eac6a None admin] RBAC: Authorizing `identity:get_user(user_id=fd491fe622e34c4aba04353a67b81134)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:58.016100 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2cc4721-67b4-484f-920e-a341708eac6a None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:58.017683 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 165/327] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:30:57 2022] GET /identity/v3/users/fd491fe622e34c4aba04353a67b81134 => generated 293 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:58.018479 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2cc4721-67b4-484f-920e-a341708eac6a None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:58.036847 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33c324b-664c-4749-8a61-977027e46966 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:58.037799 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33c324b-664c-4749-8a61-977027e46966 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:58.038221 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33c324b-664c-4749-8a61-977027e46966 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:58.038928 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33c324b-664c-4749-8a61-977027e46966 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:58.050555 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d33c324b-664c-4749-8a61-977027e46966 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:30:58.057160 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-d33c324b-664c-4749-8a61-977027e46966 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 09:30:58.057160 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:30:58.057160 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:30:58.057160 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:30:58.057160 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:30:58.057160 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:30:58.057160 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:30:58.057160 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:30:58.057160 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:30:58.057160 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:30:58.057160 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:30:58.057160 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:30:58.057160 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:30:58.057160 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:30:58.057160 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:30:58.057160 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:30:58.058485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:30:58.058485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:30:58.058485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:30:58.058485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:30:58.058485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:30:58.058485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:30:58.058485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:30:58.058485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:30:58.058485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:30:58.058485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:30:58.058485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:30:58.058485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:30:58.058485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:30:58.058485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:30:58.058485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:30:58.058485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:30:58.059394 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:30:58.059394 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:30:58.059394 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:30:58.059394 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:30:58.059394 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:30:58.059394 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:30:58.059394 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:30:58.059394 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:30:58.059394 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:30:58.059394 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:30:58.059394 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 09:30:58.059394 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:30:58.059394 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 163/328] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:30:57 2022] GET /identity/v3/roles/reader => generated 84 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:30:58.059394 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33c324b-664c-4749-8a61-977027e46966 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:58.074101 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:58.121542 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:58.129128 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2cc4721-67b4-484f-920e-a341708eac6a None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:58.167577 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84551926-3702-4bd0-b9c1-dabbeb51ece8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:58.168208 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84551926-3702-4bd0-b9c1-dabbeb51ece8 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:58.168492 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84551926-3702-4bd0-b9c1-dabbeb51ece8 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:58.168771 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84551926-3702-4bd0-b9c1-dabbeb51ece8 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:58.169879 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84551926-3702-4bd0-b9c1-dabbeb51ece8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:58.171334 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84551926-3702-4bd0-b9c1-dabbeb51ece8 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:58.172913 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 166/329] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:30:58 2022] GET /identity/v3/domains/default => generated 199 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:58.173687 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84551926-3702-4bd0-b9c1-dabbeb51ece8 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:58.193215 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33c324b-664c-4749-8a61-977027e46966 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:58.237254 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f49d373-618c-42c2-8138-0752fafc04a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:58.237629 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f49d373-618c-42c2-8138-0752fafc04a9 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:58.237908 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f49d373-618c-42c2-8138-0752fafc04a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:58.238177 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f49d373-618c-42c2-8138-0752fafc04a9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:58.240624 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f49d373-618c-42c2-8138-0752fafc04a9 None admin] RBAC: Adding query filter params (role.id=8943f289c9fe4feba2695b9c90cf3d51, user.id=fd491fe622e34c4aba04353a67b81134, scope.system=all) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:58.240624 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f49d373-618c-42c2-8138-0752fafc04a9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:58.240813 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f49d373-618c-42c2-8138-0752fafc04a9 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:58.254844 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 164/330] 10.222.0.25 () {56 vars in 1308 bytes} [Fri Oct 28 09:30:58 2022] GET /identity/v3/role_assignments?user.id=fd491fe622e34c4aba04353a67b81134&scope.system=all&role.id=8943f289c9fe4feba2695b9c90cf3d51 => generated 510 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:58.255689 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f49d373-618c-42c2-8138-0752fafc04a9 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:58.294167 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84551926-3702-4bd0-b9c1-dabbeb51ece8 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:58.346905 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5d992b-98ca-4dc8-9a1a-b72fc986d460 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:58.346905 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5d992b-98ca-4dc8-9a1a-b72fc986d460 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:58.347253 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5d992b-98ca-4dc8-9a1a-b72fc986d460 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:58.347575 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5d992b-98ca-4dc8-9a1a-b72fc986d460 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:58.348391 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a5d992b-98ca-4dc8-9a1a-b72fc986d460 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:30:58.348541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a5d992b-98ca-4dc8-9a1a-b72fc986d460 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:58.349891 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a5d992b-98ca-4dc8-9a1a-b72fc986d460 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:58.357544 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:58.361557 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 167/331] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Oct 28 09:30:58 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:30:58.367630 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5d992b-98ca-4dc8-9a1a-b72fc986d460 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:58.399363 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:58.412221 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f49d373-618c-42c2-8138-0752fafc04a9 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:58.463502 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6255a8f4-b81f-40cf-b983-03b99503f0b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:58.466446 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6255a8f4-b81f-40cf-b983-03b99503f0b1 None admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:58.466446 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6255a8f4-b81f-40cf-b983-03b99503f0b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:58.466446 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6255a8f4-b81f-40cf-b983-03b99503f0b1 None admin] PATH_INFO: `/v3/groups` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:58.466446 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6255a8f4-b81f-40cf-b983-03b99503f0b1 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:58.470320 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6255a8f4-b81f-40cf-b983-03b99503f0b1 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:58.502690 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 165/332] 10.222.0.25 () {60 vars in 1134 bytes} [Fri Oct 28 09:30:58 2022] POST /identity/v3/groups => generated 226 bytes in 247 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:30:58.522225 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5d992b-98ca-4dc8-9a1a-b72fc986d460 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:58.568939 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e6f0b2-4bb0-4a4f-afff-f50057d373aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:58.569713 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e6f0b2-4bb0-4a4f-afff-f50057d373aa None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:58.570135 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e6f0b2-4bb0-4a4f-afff-f50057d373aa None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:58.570624 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e6f0b2-4bb0-4a4f-afff-f50057d373aa None admin] PATH_INFO: `/v3/users/c8b321012884465881f5e564d2d76d7b` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:58.574751 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e6f0b2-4bb0-4a4f-afff-f50057d373aa None admin] RBAC: Authorizing `identity:get_user(user_id=c8b321012884465881f5e564d2d76d7b)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:58.574960 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32e6f0b2-4bb0-4a4f-afff-f50057d373aa None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:58.577734 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 168/333] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:30:58 2022] GET /identity/v3/users/c8b321012884465881f5e564d2d76d7b => generated 293 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:30:58.583309 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6255a8f4-b81f-40cf-b983-03b99503f0b1 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:30:58.707917 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6255a8f4-b81f-40cf-b983-03b99503f0b1 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:30:58.736273 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:30:58.749104 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb53a2dd-78bb-4f4b-a74c-152f56c31e52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:30:58.749856 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb53a2dd-78bb-4f4b-a74c-152f56c31e52 None admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:30:58.750062 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb53a2dd-78bb-4f4b-a74c-152f56c31e52 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:30:58.750339 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb53a2dd-78bb-4f4b-a74c-152f56c31e52 None admin] PATH_INFO: `/v3/system/users/c8b321012884465881f5e564d2d76d7b/roles/12f06a0e43d345aba46a51c26c012ce3` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:30:58.751842 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb53a2dd-78bb-4f4b-a74c-152f56c31e52 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=c8b321012884465881f5e564d2d76d7b, role_id=12f06a0e43d345aba46a51c26c012ce3)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:30:58.753823 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb53a2dd-78bb-4f4b-a74c-152f56c31e52 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:30:58.762473 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 166/334] 10.222.0.25 () {58 vars in 1334 bytes} [Fri Oct 28 09:30:58 2022] PUT /identity/v3/system/users/c8b321012884465881f5e564d2d76d7b/roles/12f06a0e43d345aba46a51c26c012ce3 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:31:01.488971 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8b63a0-c551-464a-8c66-7fcf02a31fee None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:01.489573 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8b63a0-c551-464a-8c66-7fcf02a31fee None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:01.489952 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8b63a0-c551-464a-8c66-7fcf02a31fee None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:01.492878 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 169/335] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:01 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:01.499028 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba035b68-92bc-4a99-88e2-0aefded57c0f None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:01.499579 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba035b68-92bc-4a99-88e2-0aefded57c0f None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:01.499997 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba035b68-92bc-4a99-88e2-0aefded57c0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:01.539733 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-ba035b68-92bc-4a99-88e2-0aefded57c0f None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:01.541715 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-ba035b68-92bc-4a99-88e2-0aefded57c0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:01.628973 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b6cacc-ae88-4c3a-9d5a-5b891f3d2b2c None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:01.629432 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b6cacc-ae88-4c3a-9d5a-5b891f3d2b2c None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:01.629885 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b6cacc-ae88-4c3a-9d5a-5b891f3d2b2c None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:01.635531 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 170/336] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:01 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:01.641023 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015a5dad-1251-42e1-95b3-9d373ed06db9 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:01.641319 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015a5dad-1251-42e1-95b3-9d373ed06db9 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:01.641576 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015a5dad-1251-42e1-95b3-9d373ed06db9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:01.683492 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-015a5dad-1251-42e1-95b3-9d373ed06db9 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:01.684705 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-015a5dad-1251-42e1-95b3-9d373ed06db9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:01.701188 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 167/337] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:31:01.707016 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d9e750-963c-4943-a21f-2520595c0a48 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:01.707763 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d9e750-963c-4943-a21f-2520595c0a48 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:01.708244 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d9e750-963c-4943-a21f-2520595c0a48 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:01.710043 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 168/338] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:01.728804 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00458ca0-ccee-4678-8fcc-8d39f6be42a8 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:01.729437 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00458ca0-ccee-4678-8fcc-8d39f6be42a8 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:01.729915 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00458ca0-ccee-4678-8fcc-8d39f6be42a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:01.759561 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-00458ca0-ccee-4678-8fcc-8d39f6be42a8 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:01.761128 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-00458ca0-ccee-4678-8fcc-8d39f6be42a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:01.837697 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 171/339] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 09:31:01.871112 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 169/340] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:31:02.103865 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-015a5dad-1251-42e1-95b3-9d373ed06db9 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:02.159740 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:02.272953 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00458ca0-ccee-4678-8fcc-8d39f6be42a8 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:02.354972 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:02.360797 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-015a5dad-1251-42e1-95b3-9d373ed06db9 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:02.445911 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00458ca0-ccee-4678-8fcc-8d39f6be42a8 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:02.458770 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48901acc-d2f5-4332-869d-b17064f94af5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:02.459763 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48901acc-d2f5-4332-869d-b17064f94af5 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:02.460246 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48901acc-d2f5-4332-869d-b17064f94af5 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:02.461250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48901acc-d2f5-4332-869d-b17064f94af5 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:02.483163 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48901acc-d2f5-4332-869d-b17064f94af5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:31:02.494438 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-48901acc-d2f5-4332-869d-b17064f94af5 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 09:31:02.494438 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:02.494438 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:02.494438 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:02.494438 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:02.494438 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:02.494438 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:02.494438 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:02.494438 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:02.494438 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:02.494438 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:02.494438 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:02.494438 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:31:02.494438 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:31:02.494438 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:31:02.494438 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:31:02.495850 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:31:02.495850 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:31:02.495850 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:31:02.495850 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:31:02.495850 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:02.495850 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:02.495850 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:02.495850 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:02.495850 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:02.495850 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:02.495850 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:02.495850 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:02.495850 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:02.495850 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:02.495850 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:02.495850 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:02.496653 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:02.496653 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:02.496653 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:02.496653 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:02.496653 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:31:02.496653 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:31:02.496653 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:31:02.496653 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:31:02.496653 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:31:02.496653 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:31:02.496653 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 09:31:02.496653 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:02.502258 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 172/341] 10.222.0.25 () {56 vars in 1114 bytes} [Fri Oct 28 09:31:02 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 399 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:02.504411 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0193b16-1dcb-4f13-8826-f8ee6668ad71 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:02.507387 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0193b16-1dcb-4f13-8826-f8ee6668ad71 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:02.507805 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0193b16-1dcb-4f13-8826-f8ee6668ad71 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:02.509585 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 173/342] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:02 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:02.510671 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0193b16-1dcb-4f13-8826-f8ee6668ad71 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:02.533575 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-590ecc24-5043-4b59-8a6c-3189043513f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:02.534820 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590ecc24-5043-4b59-8a6c-3189043513f1 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:02.535312 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590ecc24-5043-4b59-8a6c-3189043513f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:02.535868 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590ecc24-5043-4b59-8a6c-3189043513f1 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:02.583405 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-590ecc24-5043-4b59-8a6c-3189043513f1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:31:02.593857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-590ecc24-5043-4b59-8a6c-3189043513f1 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 09:31:02.593857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:02.593857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:02.593857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:02.593857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:02.593857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:02.593857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:02.593857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:02.593857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:02.593857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:02.593857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:02.593857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:02.593857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:31:02.593857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:31:02.593857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:31:02.593857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:31:02.594842 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:31:02.594842 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:31:02.594842 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:31:02.594842 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:31:02.594842 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:02.594842 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:02.594842 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:02.594842 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:02.594842 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:02.594842 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:02.594842 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:02.594842 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:02.594842 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:02.594842 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:02.594842 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:02.594842 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:02.595618 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:02.595618 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:02.595618 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:02.595618 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:02.595618 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:31:02.595618 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:31:02.595618 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:31:02.595618 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:31:02.595618 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:31:02.595618 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:31:02.595618 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 09:31:02.595618 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:02.600208 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 170/343] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:31:02 2022] GET /identity/v3/roles/member => generated 84 bytes in 328 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:02.607833 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dcb544f-7714-414e-aaf7-a1d277c108c4 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:02.607996 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dcb544f-7714-414e-aaf7-a1d277c108c4 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:02.608452 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dcb544f-7714-414e-aaf7-a1d277c108c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:02.634798 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:02.640100 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0193b16-1dcb-4f13-8826-f8ee6668ad71 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:02.655641 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-0dcb544f-7714-414e-aaf7-a1d277c108c4 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:02.658036 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-0dcb544f-7714-414e-aaf7-a1d277c108c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:02.680613 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955c46f6-f340-4d9d-828d-313ee436bf32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:02.681955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955c46f6-f340-4d9d-828d-313ee436bf32 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:02.682441 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955c46f6-f340-4d9d-828d-313ee436bf32 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:02.683101 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955c46f6-f340-4d9d-828d-313ee436bf32 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:02.684007 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-955c46f6-f340-4d9d-828d-313ee436bf32 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:02.684479 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-955c46f6-f340-4d9d-828d-313ee436bf32 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:02.686108 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-955c46f6-f340-4d9d-828d-313ee436bf32 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:02.697851 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 174/344] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:31:02 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:02.699005 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955c46f6-f340-4d9d-828d-313ee436bf32 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:02.803749 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 171/345] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:31:02.805031 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dcb544f-7714-414e-aaf7-a1d277c108c4 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:02.854718 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:02.873498 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955c46f6-f340-4d9d-828d-313ee436bf32 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:02.934024 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dcb544f-7714-414e-aaf7-a1d277c108c4 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:02.945494 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8be297c-1e92-407f-b680-27c8f3fbd24e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:02.946563 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8be297c-1e92-407f-b680-27c8f3fbd24e None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:02.947678 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8be297c-1e92-407f-b680-27c8f3fbd24e None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:02.948129 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8be297c-1e92-407f-b680-27c8f3fbd24e None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:02.949269 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8be297c-1e92-407f-b680-27c8f3fbd24e None admin] RBAC: Adding query filter params (name=member) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:02.949774 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8be297c-1e92-407f-b680-27c8f3fbd24e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:02.954173 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8be297c-1e92-407f-b680-27c8f3fbd24e None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:02.977372 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 175/346] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Oct 28 09:31:02 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:02.977372 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8be297c-1e92-407f-b680-27c8f3fbd24e None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:02.986055 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea830c1-ca58-4498-811f-358bdba7723f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:02.987076 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea830c1-ca58-4498-811f-358bdba7723f None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:02.987573 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea830c1-ca58-4498-811f-358bdba7723f None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:02.988034 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea830c1-ca58-4498-811f-358bdba7723f None admin] PATH_INFO: `/v3/roles/member` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:02.996751 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:03.001413 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea830c1-ca58-4498-811f-358bdba7723f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:31:03.008233 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-cea830c1-ca58-4498-811f-358bdba7723f None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 09:31:03.008233 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:03.008233 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:03.008233 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:03.008233 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:03.008233 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:03.008233 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:03.008233 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:03.008233 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:03.008233 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:03.008233 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:03.008233 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:03.008233 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:31:03.008233 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:31:03.008233 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:31:03.008233 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:31:03.009244 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:31:03.009244 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:31:03.009244 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:31:03.009244 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:31:03.009244 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:03.009244 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:03.009244 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:03.009244 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:03.009244 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:03.009244 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:03.009244 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:03.009244 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:03.009244 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:03.009244 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:03.009244 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:03.009244 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:03.010047 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:03.010047 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:03.010047 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:03.010047 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:03.010047 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:31:03.010047 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:31:03.010047 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:31:03.010047 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:31:03.010047 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:31:03.010047 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:31:03.010047 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 09:31:03.010047 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:03.012506 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 172/347] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:31:02 2022] GET /identity/v3/roles/member => generated 84 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:03.013557 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea830c1-ca58-4498-811f-358bdba7723f None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:03.114765 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8be297c-1e92-407f-b680-27c8f3fbd24e None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:03.130657 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:03.136284 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea830c1-ca58-4498-811f-358bdba7723f None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:03.153362 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203f26b1-d2e7-4473-86a9-8a18a7d29b3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:03.153977 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203f26b1-d2e7-4473-86a9-8a18a7d29b3f None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:03.154256 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203f26b1-d2e7-4473-86a9-8a18a7d29b3f None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:03.154560 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203f26b1-d2e7-4473-86a9-8a18a7d29b3f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:03.178930 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17845e3-4511-45f6-9350-4662fc0bf3bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:03.180452 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17845e3-4511-45f6-9350-4662fc0bf3bc None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:03.181036 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17845e3-4511-45f6-9350-4662fc0bf3bc None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:03.181536 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17845e3-4511-45f6-9350-4662fc0bf3bc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:03.232691 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-203f26b1-d2e7-4473-86a9-8a18a7d29b3f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:03.234129 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-203f26b1-d2e7-4473-86a9-8a18a7d29b3f None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:03.243517 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:03.258829 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f17845e3-4511-45f6-9350-4662fc0bf3bc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:03.260650 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f17845e3-4511-45f6-9350-4662fc0bf3bc None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:03.354487 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 176/348] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:31:02 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 378 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 09:31:03.355696 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203f26b1-d2e7-4473-86a9-8a18a7d29b3f None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:03.372733 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:03.379164 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 173/349] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:31:03 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 366 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 09:31:03.380275 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17845e3-4511-45f6-9350-4662fc0bf3bc None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:03.481051 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203f26b1-d2e7-4473-86a9-8a18a7d29b3f None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:03.506348 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17845e3-4511-45f6-9350-4662fc0bf3bc None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:03.518877 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:03.531860 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c22420-4d28-4797-8ace-f4767d4d05a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:03.532737 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c22420-4d28-4797-8ace-f4767d4d05a0 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:03.533015 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c22420-4d28-4797-8ace-f4767d4d05a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:03.533285 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c22420-4d28-4797-8ace-f4767d4d05a0 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:03.533961 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54c22420-4d28-4797-8ace-f4767d4d05a0 None admin] RBAC: Adding query filter params (name=member) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:03.534265 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54c22420-4d28-4797-8ace-f4767d4d05a0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:03.535676 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54c22420-4d28-4797-8ace-f4767d4d05a0 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:03.550945 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 177/350] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Oct 28 09:31:03 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:03.552782 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c22420-4d28-4797-8ace-f4767d4d05a0 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:03.561191 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-991a5a04-833a-4491-ae49-67f4556b4ef7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:03.562950 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991a5a04-833a-4491-ae49-67f4556b4ef7 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:03.562950 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991a5a04-833a-4491-ae49-67f4556b4ef7 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:03.562950 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991a5a04-833a-4491-ae49-67f4556b4ef7 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:03.581157 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-991a5a04-833a-4491-ae49-67f4556b4ef7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:31:03.585417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-991a5a04-833a-4491-ae49-67f4556b4ef7 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 09:31:03.585417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:03.585417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:03.585417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:03.585417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:03.585417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:03.585417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:03.585417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:03.585417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:03.585417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:03.585417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:03.585417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:03.585417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:31:03.585417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:31:03.585417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:31:03.585417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:31:03.586580 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:31:03.586580 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:31:03.586580 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:31:03.586580 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:31:03.586580 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:03.586580 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:03.586580 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:03.586580 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:03.586580 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:03.586580 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:03.586580 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:03.586580 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:03.586580 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:03.586580 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:03.586580 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:03.586580 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:03.587404 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:03.587404 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:03.587404 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:03.587404 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:03.587404 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:31:03.587404 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:31:03.587404 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:31:03.587404 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:31:03.587404 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:31:03.587404 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:31:03.587404 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Oct 28 09:31:03.587404 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:03.587404 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 174/351] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:31:03 2022] GET /identity/v3/roles/reader => generated 84 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:03.588206 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-991a5a04-833a-4491-ae49-67f4556b4ef7 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:03.672198 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:03.705801 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c22420-4d28-4797-8ace-f4767d4d05a0 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:03.736915 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-991a5a04-833a-4491-ae49-67f4556b4ef7 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:03.763807 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109f0d15-c7dc-4320-9501-2e942f4081b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:03.764781 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109f0d15-c7dc-4320-9501-2e942f4081b1 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:03.764957 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109f0d15-c7dc-4320-9501-2e942f4081b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:03.765326 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109f0d15-c7dc-4320-9501-2e942f4081b1 None admin] PATH_INFO: `/v3/projects/6e28083c55ff4290af640f011fd2466f` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:03.766712 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-109f0d15-c7dc-4320-9501-2e942f4081b1 None admin] RBAC: Authorizing `identity:get_project(project_id=6e28083c55ff4290af640f011fd2466f)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:03.768440 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-109f0d15-c7dc-4320-9501-2e942f4081b1 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:03.774497 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 178/352] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:31:03 2022] GET /identity/v3/projects/6e28083c55ff4290af640f011fd2466f => generated 298 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:03.775839 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109f0d15-c7dc-4320-9501-2e942f4081b1 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:03.791762 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71512892-bc31-47d2-bcc9-2b9ee03638c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:03.793326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71512892-bc31-47d2-bcc9-2b9ee03638c4 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:03.793617 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71512892-bc31-47d2-bcc9-2b9ee03638c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:03.794189 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71512892-bc31-47d2-bcc9-2b9ee03638c4 None admin] PATH_INFO: `/v3/projects/5960af42464f4e83b60ec52264d04b70` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:03.795500 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71512892-bc31-47d2-bcc9-2b9ee03638c4 None admin] RBAC: Authorizing `identity:get_project(project_id=5960af42464f4e83b60ec52264d04b70)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:03.797970 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71512892-bc31-47d2-bcc9-2b9ee03638c4 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:03.800008 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 175/353] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:31:03 2022] GET /identity/v3/projects/5960af42464f4e83b60ec52264d04b70 => generated 302 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:03.801585 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343e1944-60b2-4f94-9fb9-36d703633171 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:03.801873 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343e1944-60b2-4f94-9fb9-36d703633171 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:03.802155 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-343e1944-60b2-4f94-9fb9-36d703633171 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:03.804024 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 176/354] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:03.805664 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-343e1944-60b2-4f94-9fb9-36d703633171 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:03.822535 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:03.917708 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109f0d15-c7dc-4320-9501-2e942f4081b1 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:03.967934 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-343e1944-60b2-4f94-9fb9-36d703633171 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:03.969590 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a17a3c15-1595-4bc3-88a6-b4ce906869c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:03.970240 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a17a3c15-1595-4bc3-88a6-b4ce906869c7 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:03.970552 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a17a3c15-1595-4bc3-88a6-b4ce906869c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:03.970833 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a17a3c15-1595-4bc3-88a6-b4ce906869c7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:04.010798 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:04.022810 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c3fd423-30e8-4555-baf4-861dba22c193 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:04.024864 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3fd423-30e8-4555-baf4-861dba22c193 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:04.025464 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3fd423-30e8-4555-baf4-861dba22c193 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:04.025955 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3fd423-30e8-4555-baf4-861dba22c193 None admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:04.026978 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c3fd423-30e8-4555-baf4-861dba22c193 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:04.027508 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c3fd423-30e8-4555-baf4-861dba22c193 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:04.029192 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c3fd423-30e8-4555-baf4-861dba22c193 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:04.040812 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 177/355] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Oct 28 09:31:03 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:04.042308 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c3fd423-30e8-4555-baf4-861dba22c193 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:04.069501 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a17a3c15-1595-4bc3-88a6-b4ce906869c7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:04.071715 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a17a3c15-1595-4bc3-88a6-b4ce906869c7 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:04.115197 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:04.177657 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c3fd423-30e8-4555-baf4-861dba22c193 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:04.207819 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 179/356] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:31:03 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 433 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 09:31:04.208695 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a17a3c15-1595-4bc3-88a6-b4ce906869c7 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:04.218574 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0018fab9-3e49-40f9-ab2d-887d507555d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:04.219239 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0018fab9-3e49-40f9-ab2d-887d507555d2 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:04.219958 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0018fab9-3e49-40f9-ab2d-887d507555d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:04.219958 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0018fab9-3e49-40f9-ab2d-887d507555d2 None admin] PATH_INFO: `/v3/groups/71818c8c50704a6ca8dfade2fa6e5a7d` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:04.226955 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0018fab9-3e49-40f9-ab2d-887d507555d2 None admin] RBAC: Authorizing `identity:get_group(group_id=71818c8c50704a6ca8dfade2fa6e5a7d)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:04.228468 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0018fab9-3e49-40f9-ab2d-887d507555d2 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:04.230167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 178/357] 10.222.0.25 () {56 vars in 1180 bytes} [Fri Oct 28 09:31:04 2022] GET /identity/v3/groups/71818c8c50704a6ca8dfade2fa6e5a7d => generated 226 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:04.231394 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ef37bf-c324-441e-a104-cc5ff0b01eaa None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:04.231666 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ef37bf-c324-441e-a104-cc5ff0b01eaa None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:04.231943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ef37bf-c324-441e-a104-cc5ff0b01eaa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:04.250753 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-77ef37bf-c324-441e-a104-cc5ff0b01eaa None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:04.251982 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-77ef37bf-c324-441e-a104-cc5ff0b01eaa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:04.315342 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:04.333623 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a17a3c15-1595-4bc3-88a6-b4ce906869c7 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:04.341934 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 179/358] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:31:04.344239 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ef37bf-c324-441e-a104-cc5ff0b01eaa None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:04.384085 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b4fb025-700f-4589-9ad3-4ecb9a17b496 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:04.384775 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4fb025-700f-4589-9ad3-4ecb9a17b496 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:04.385048 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4fb025-700f-4589-9ad3-4ecb9a17b496 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:04.385322 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b4fb025-700f-4589-9ad3-4ecb9a17b496 None admin] PATH_INFO: `/v3/groups/71818c8c50704a6ca8dfade2fa6e5a7d` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:04.387373 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b4fb025-700f-4589-9ad3-4ecb9a17b496 None admin] RBAC: Authorizing `identity:get_group(group_id=71818c8c50704a6ca8dfade2fa6e5a7d)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:04.388788 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b4fb025-700f-4589-9ad3-4ecb9a17b496 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:04.391618 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 180/359] 10.222.0.25 () {56 vars in 1180 bytes} [Fri Oct 28 09:31:04 2022] GET /identity/v3/groups/71818c8c50704a6ca8dfade2fa6e5a7d => generated 226 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:04.391755 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6d0357-02f2-4fb0-bbdd-005a4b4f6cd2 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:04.391991 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6d0357-02f2-4fb0-bbdd-005a4b4f6cd2 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:04.393608 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6d0357-02f2-4fb0-bbdd-005a4b4f6cd2 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:04.393608 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 181/360] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:04 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:04.394427 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6d0357-02f2-4fb0-bbdd-005a4b4f6cd2 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:04.404634 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:04.486175 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ef37bf-c324-441e-a104-cc5ff0b01eaa None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:04.532871 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c563e762-0b2e-4d5b-803a-fba948d22f80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:04.533522 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c563e762-0b2e-4d5b-803a-fba948d22f80 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:04.534094 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c563e762-0b2e-4d5b-803a-fba948d22f80 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:04.534094 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c563e762-0b2e-4d5b-803a-fba948d22f80 None admin] PATH_INFO: `/v3/users/c8b321012884465881f5e564d2d76d7b` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:04.535283 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c563e762-0b2e-4d5b-803a-fba948d22f80 None admin] RBAC: Authorizing `identity:get_user(user_id=c8b321012884465881f5e564d2d76d7b)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:04.536735 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c563e762-0b2e-4d5b-803a-fba948d22f80 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:04.538330 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 180/361] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:31:04 2022] GET /identity/v3/users/c8b321012884465881f5e564d2d76d7b => generated 293 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:04.539669 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c563e762-0b2e-4d5b-803a-fba948d22f80 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:04.541740 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6d0357-02f2-4fb0-bbdd-005a4b4f6cd2 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:04.592997 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eee7bee-90f3-4d02-b074-86e6a3f11ab0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:04.593674 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eee7bee-90f3-4d02-b074-86e6a3f11ab0 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:04.593955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eee7bee-90f3-4d02-b074-86e6a3f11ab0 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:04.594227 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eee7bee-90f3-4d02-b074-86e6a3f11ab0 None admin] PATH_INFO: `/v3/projects/6e28083c55ff4290af640f011fd2466f` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:04.595457 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eee7bee-90f3-4d02-b074-86e6a3f11ab0 None admin] RBAC: Authorizing `identity:get_project(project_id=6e28083c55ff4290af640f011fd2466f)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:04.596924 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eee7bee-90f3-4d02-b074-86e6a3f11ab0 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:04.599027 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 182/362] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:31:04 2022] GET /identity/v3/projects/6e28083c55ff4290af640f011fd2466f => generated 298 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:04.599896 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eee7bee-90f3-4d02-b074-86e6a3f11ab0 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:04.621278 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:04.681260 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c563e762-0b2e-4d5b-803a-fba948d22f80 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:04.691829 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:04.720025 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd998b1-f636-4a43-9dc2-0e003b986298 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:04.720642 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd998b1-f636-4a43-9dc2-0e003b986298 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:04.720924 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd998b1-f636-4a43-9dc2-0e003b986298 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:04.721200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd998b1-f636-4a43-9dc2-0e003b986298 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:04.721926 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd998b1-f636-4a43-9dc2-0e003b986298 None admin] RBAC: Adding query filter params (group.id=71818c8c50704a6ca8dfade2fa6e5a7d, role.id=d93b4c94b590457199c3d301f947b0fb, scope.project.id=6e28083c55ff4290af640f011fd2466f) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:04.722238 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd998b1-f636-4a43-9dc2-0e003b986298 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:04.723714 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd998b1-f636-4a43-9dc2-0e003b986298 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:04.727671 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eee7bee-90f3-4d02-b074-86e6a3f11ab0 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:04.731081 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 181/363] 10.222.0.25 () {56 vars in 1376 bytes} [Fri Oct 28 09:31:04 2022] GET /identity/v3/role_assignments?group.id=71818c8c50704a6ca8dfade2fa6e5a7d&scope.project.id=6e28083c55ff4290af640f011fd2466f&role.id=d93b4c94b590457199c3d301f947b0fb => generated 260 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:04.732156 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3f3945-502e-438a-a685-2f9e3bc58108 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:04.732444 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3f3945-502e-438a-a685-2f9e3bc58108 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:04.732718 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3f3945-502e-438a-a685-2f9e3bc58108 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:04.753372 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-ea3f3945-502e-438a-a685-2f9e3bc58108 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:04.755867 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-ea3f3945-502e-438a-a685-2f9e3bc58108 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:04.776232 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4619e85a-a22c-4281-8b31-832f531d6f7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:04.776863 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4619e85a-a22c-4281-8b31-832f531d6f7b None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:04.777145 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4619e85a-a22c-4281-8b31-832f531d6f7b None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:04.777419 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4619e85a-a22c-4281-8b31-832f531d6f7b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:04.778172 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4619e85a-a22c-4281-8b31-832f531d6f7b None admin] RBAC: Adding query filter params (group.id=71818c8c50704a6ca8dfade2fa6e5a7d, role.id=8943f289c9fe4feba2695b9c90cf3d51, scope.project.id=5960af42464f4e83b60ec52264d04b70) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:04.778509 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4619e85a-a22c-4281-8b31-832f531d6f7b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:04.779984 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4619e85a-a22c-4281-8b31-832f531d6f7b None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:04.789950 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 183/364] 10.222.0.25 () {56 vars in 1376 bytes} [Fri Oct 28 09:31:04 2022] GET /identity/v3/role_assignments?group.id=71818c8c50704a6ca8dfade2fa6e5a7d&scope.project.id=5960af42464f4e83b60ec52264d04b70&role.id=8943f289c9fe4feba2695b9c90cf3d51 => generated 260 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:04.789950 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4619e85a-a22c-4281-8b31-832f531d6f7b None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:04.885475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 182/365] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:31:04.890496 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea3f3945-502e-438a-a685-2f9e3bc58108 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:04.973318 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4619e85a-a22c-4281-8b31-832f531d6f7b None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:04.985453 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:05.021595 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422722ec-c535-4005-ad5f-13da5dc40d43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:05.022218 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422722ec-c535-4005-ad5f-13da5dc40d43 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:05.022533 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422722ec-c535-4005-ad5f-13da5dc40d43 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:05.022812 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422722ec-c535-4005-ad5f-13da5dc40d43 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:05.023626 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-422722ec-c535-4005-ad5f-13da5dc40d43 None admin] RBAC: Adding query filter params (role.id=12f06a0e43d345aba46a51c26c012ce3, user.id=c8b321012884465881f5e564d2d76d7b, scope.system=all) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:05.023942 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-422722ec-c535-4005-ad5f-13da5dc40d43 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:05.025379 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-422722ec-c535-4005-ad5f-13da5dc40d43 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:05.029090 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:05.043016 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 184/366] 10.222.0.25 () {56 vars in 1308 bytes} [Fri Oct 28 09:31:04 2022] GET /identity/v3/role_assignments?user.id=c8b321012884465881f5e564d2d76d7b&scope.system=all&role.id=12f06a0e43d345aba46a51c26c012ce3 => generated 510 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:05.043907 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422722ec-c535-4005-ad5f-13da5dc40d43 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:05.056513 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea3f3945-502e-438a-a685-2f9e3bc58108 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:05.099424 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49560c3-cb90-4bbb-826b-6e5954229543 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:05.100480 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49560c3-cb90-4bbb-826b-6e5954229543 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:05.100764 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49560c3-cb90-4bbb-826b-6e5954229543 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:05.101078 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49560c3-cb90-4bbb-826b-6e5954229543 None admin] PATH_INFO: `/v3/groups/71818c8c50704a6ca8dfade2fa6e5a7d` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:05.102818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f49560c3-cb90-4bbb-826b-6e5954229543 None admin] RBAC: Authorizing `identity:get_group(group_id=71818c8c50704a6ca8dfade2fa6e5a7d)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:05.104262 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f49560c3-cb90-4bbb-826b-6e5954229543 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:05.106933 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 183/367] 10.222.0.25 () {56 vars in 1180 bytes} [Fri Oct 28 09:31:04 2022] GET /identity/v3/groups/71818c8c50704a6ca8dfade2fa6e5a7d => generated 226 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:05.113444 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49560c3-cb90-4bbb-826b-6e5954229543 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:05.262084 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422722ec-c535-4005-ad5f-13da5dc40d43 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:05.301497 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea29efdd-3167-4e6e-adde-4917f0407fc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:05.302380 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea29efdd-3167-4e6e-adde-4917f0407fc1 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:05.302828 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea29efdd-3167-4e6e-adde-4917f0407fc1 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:05.303240 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea29efdd-3167-4e6e-adde-4917f0407fc1 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:05.315677 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea29efdd-3167-4e6e-adde-4917f0407fc1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:31:05.321787 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-ea29efdd-3167-4e6e-adde-4917f0407fc1 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 09:31:05.321787 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:05.321787 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:05.321787 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:05.321787 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:05.321787 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:05.321787 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:05.321787 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:05.321787 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:05.321787 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:05.321787 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:05.321787 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:05.321787 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:31:05.321787 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:31:05.321787 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:31:05.321787 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:31:05.324713 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:31:05.324713 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:31:05.324713 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:31:05.324713 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:31:05.324713 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:05.324713 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:05.324713 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:05.324713 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:05.324713 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:05.324713 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:05.324713 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:05.324713 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:05.324713 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:05.324713 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:05.324713 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:05.324713 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:05.325583 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:05.325583 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:05.325583 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:05.325583 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:05.325583 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:31:05.325583 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:31:05.325583 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:31:05.325583 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:31:05.325583 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:31:05.325583 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:31:05.325583 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 09:31:05.325583 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:05.328200 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 185/368] 10.222.0.25 () {56 vars in 1114 bytes} [Fri Oct 28 09:31:05 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 284 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:05.328993 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49560c3-cb90-4bbb-826b-6e5954229543 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:05.329367 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea29efdd-3167-4e6e-adde-4917f0407fc1 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:05.353202 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:05.387146 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af37fabf-0838-45cd-8a26-dc6b6643454e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:05.388030 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af37fabf-0838-45cd-8a26-dc6b6643454e None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:05.388459 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af37fabf-0838-45cd-8a26-dc6b6643454e None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:05.388904 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af37fabf-0838-45cd-8a26-dc6b6643454e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:05.389779 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af37fabf-0838-45cd-8a26-dc6b6643454e None admin] RBAC: Adding query filter params (group.id=71818c8c50704a6ca8dfade2fa6e5a7d, role.id=8943f289c9fe4feba2695b9c90cf3d51, scope.project.id=6e28083c55ff4290af640f011fd2466f) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:05.390236 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af37fabf-0838-45cd-8a26-dc6b6643454e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:05.392667 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af37fabf-0838-45cd-8a26-dc6b6643454e None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:05.404433 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 184/369] 10.222.0.25 () {56 vars in 1376 bytes} [Fri Oct 28 09:31:05 2022] GET /identity/v3/role_assignments?group.id=71818c8c50704a6ca8dfade2fa6e5a7d&scope.project.id=6e28083c55ff4290af640f011fd2466f&role.id=8943f289c9fe4feba2695b9c90cf3d51 => generated 260 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:05.406077 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af37fabf-0838-45cd-8a26-dc6b6643454e None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:05.408941 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:05.472312 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea29efdd-3167-4e6e-adde-4917f0407fc1 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:05.520284 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-837eb855-b61b-4f2e-bfb1-8f8ae0927d6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:05.520931 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837eb855-b61b-4f2e-bfb1-8f8ae0927d6b None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:05.521224 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837eb855-b61b-4f2e-bfb1-8f8ae0927d6b None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:05.521504 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837eb855-b61b-4f2e-bfb1-8f8ae0927d6b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:05.545973 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-837eb855-b61b-4f2e-bfb1-8f8ae0927d6b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:31:05.553094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-837eb855-b61b-4f2e-bfb1-8f8ae0927d6b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:31:05.553094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:05.553094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:05.553094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:05.553094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:05.553094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:05.553094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:05.553094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:05.553094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:05.553094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:05.553094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:05.553094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:05.553094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:31:05.553094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:31:05.553094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:31:05.553094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:31:05.554819 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:31:05.554819 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:31:05.554819 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:31:05.554819 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:31:05.554819 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:05.554819 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:05.554819 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:05.554819 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:05.554819 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:05.554819 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:05.554819 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:05.554819 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:05.554819 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:05.554819 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:05.554819 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:05.554819 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:05.555655 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:05.555655 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:05.555655 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:05.555655 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:05.555655 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:31:05.555655 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:31:05.555655 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:31:05.555655 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:31:05.555655 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:31:05.555655 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:31:05.555655 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:31:05.555655 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:05.558005 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 186/370] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Oct 28 09:31:05 2022] GET /identity/v3/roles/admin => generated 83 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:05.564862 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af37fabf-0838-45cd-8a26-dc6b6643454e None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:05.565854 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-837eb855-b61b-4f2e-bfb1-8f8ae0927d6b None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:05.635469 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b81c9a-5d2d-4a17-859b-8a0ed6cbaf6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:05.636726 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b81c9a-5d2d-4a17-859b-8a0ed6cbaf6f None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:05.637251 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b81c9a-5d2d-4a17-859b-8a0ed6cbaf6f None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:05.637683 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b81c9a-5d2d-4a17-859b-8a0ed6cbaf6f None admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:05.638615 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3b81c9a-5d2d-4a17-859b-8a0ed6cbaf6f None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:05.639094 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3b81c9a-5d2d-4a17-859b-8a0ed6cbaf6f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:05.643214 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3b81c9a-5d2d-4a17-859b-8a0ed6cbaf6f None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:05.668435 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 185/371] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:31:05 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:05.673966 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b81c9a-5d2d-4a17-859b-8a0ed6cbaf6f None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:05.717904 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-837eb855-b61b-4f2e-bfb1-8f8ae0927d6b None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:05.734936 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:05.767797 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-684d61f5-048b-426a-a845-9795808101c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:05.768699 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684d61f5-048b-426a-a845-9795808101c4 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:05.769121 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684d61f5-048b-426a-a845-9795808101c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:05.769534 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684d61f5-048b-426a-a845-9795808101c4 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:05.770430 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-684d61f5-048b-426a-a845-9795808101c4 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:05.770896 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-684d61f5-048b-426a-a845-9795808101c4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:05.772423 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-684d61f5-048b-426a-a845-9795808101c4 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:05.775000 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:05.791600 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 187/372] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:31:05 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:31:05.797667 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-684d61f5-048b-426a-a845-9795808101c4 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:05.905508 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b81c9a-5d2d-4a17-859b-8a0ed6cbaf6f None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:05.952245 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-684d61f5-048b-426a-a845-9795808101c4 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:05.962120 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e807a070-3afc-4041-9e22-6104dd456f4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:05.963015 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e807a070-3afc-4041-9e22-6104dd456f4e None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:05.963448 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e807a070-3afc-4041-9e22-6104dd456f4e None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:05.963865 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e807a070-3afc-4041-9e22-6104dd456f4e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:05.994209 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d4cd2c3-5086-4a7e-8391-bf156ed5d5fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:05.994864 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4cd2c3-5086-4a7e-8391-bf156ed5d5fe None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:05.995148 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4cd2c3-5086-4a7e-8391-bf156ed5d5fe None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:05.995425 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4cd2c3-5086-4a7e-8391-bf156ed5d5fe None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:06.049507 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:06.074667 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d4cd2c3-5086-4a7e-8391-bf156ed5d5fe None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:06.076304 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d4cd2c3-5086-4a7e-8391-bf156ed5d5fe None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:06.080459 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e807a070-3afc-4041-9e22-6104dd456f4e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:06.082173 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e807a070-3afc-4041-9e22-6104dd456f4e None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:06.147685 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:06.189431 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 188/373] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:31:05 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 392 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 09:31:06.196319 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d4cd2c3-5086-4a7e-8391-bf156ed5d5fe None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:06.230558 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 186/374] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:31:05 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 557 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 28 09:31:06.236018 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e807a070-3afc-4041-9e22-6104dd456f4e None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:06.315430 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:06.341633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d4cd2c3-5086-4a7e-8391-bf156ed5d5fe None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:06.383797 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298e761e-e733-42d7-915d-f55d6da0d9b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:06.384709 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298e761e-e733-42d7-915d-f55d6da0d9b7 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:06.385129 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298e761e-e733-42d7-915d-f55d6da0d9b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:06.385591 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298e761e-e733-42d7-915d-f55d6da0d9b7 None admin] PATH_INFO: `/v3/projects/ac54d153685c4b2288a7ed215d0e24d2` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:06.386923 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-298e761e-e733-42d7-915d-f55d6da0d9b7 None admin] RBAC: Authorizing `identity:get_project(project_id=ac54d153685c4b2288a7ed215d0e24d2)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:06.388576 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-298e761e-e733-42d7-915d-f55d6da0d9b7 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:06.390486 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 189/375] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:31:06 2022] GET /identity/v3/projects/ac54d153685c4b2288a7ed215d0e24d2 => generated 344 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:06.396654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298e761e-e733-42d7-915d-f55d6da0d9b7 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:06.454213 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e807a070-3afc-4041-9e22-6104dd456f4e None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:06.517552 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298e761e-e733-42d7-915d-f55d6da0d9b7 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:06.529469 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-befac0dd-6644-49ff-9e41-e86f07a6a303 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:06.530121 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-befac0dd-6644-49ff-9e41-e86f07a6a303 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:06.530436 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-befac0dd-6644-49ff-9e41-e86f07a6a303 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:06.530701 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-befac0dd-6644-49ff-9e41-e86f07a6a303 None admin] PATH_INFO: `/v3/projects/5960af42464f4e83b60ec52264d04b70` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:06.533816 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-befac0dd-6644-49ff-9e41-e86f07a6a303 None admin] RBAC: Authorizing `identity:get_project(project_id=5960af42464f4e83b60ec52264d04b70)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:06.535315 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-befac0dd-6644-49ff-9e41-e86f07a6a303 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:06.537011 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 187/376] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:31:06 2022] GET /identity/v3/projects/5960af42464f4e83b60ec52264d04b70 => generated 302 bytes in 301 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:31:06.543454 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-befac0dd-6644-49ff-9e41-e86f07a6a303 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:06.560712 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:06.591226 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242cc1b4-923c-4829-8b52-f16e91ae6720 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:06.592670 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242cc1b4-923c-4829-8b52-f16e91ae6720 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:06.593154 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242cc1b4-923c-4829-8b52-f16e91ae6720 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:06.593640 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242cc1b4-923c-4829-8b52-f16e91ae6720 None admin] PATH_INFO: `/v3/groups/1c414fe7bf464383a9eee4581f8f3c2f` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:06.601823 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-242cc1b4-923c-4829-8b52-f16e91ae6720 None admin] RBAC: Authorizing `identity:get_group(group_id=1c414fe7bf464383a9eee4581f8f3c2f)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:06.603863 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-242cc1b4-923c-4829-8b52-f16e91ae6720 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:06.606657 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 190/377] 10.222.0.25 () {56 vars in 1180 bytes} [Fri Oct 28 09:31:06 2022] GET /identity/v3/groups/1c414fe7bf464383a9eee4581f8f3c2f => generated 229 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:31:06.612805 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242cc1b4-923c-4829-8b52-f16e91ae6720 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:06.625959 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:06.691632 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-befac0dd-6644-49ff-9e41-e86f07a6a303 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:06.729733 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242cc1b4-923c-4829-8b52-f16e91ae6720 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:06.750707 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4edf84c3-3cbf-484e-b190-16d46483902b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:06.751733 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4edf84c3-3cbf-484e-b190-16d46483902b None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:06.753574 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4edf84c3-3cbf-484e-b190-16d46483902b None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:06.754061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4edf84c3-3cbf-484e-b190-16d46483902b None admin] PATH_INFO: `/v3/groups/71818c8c50704a6ca8dfade2fa6e5a7d` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:06.756537 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4edf84c3-3cbf-484e-b190-16d46483902b None admin] RBAC: Authorizing `identity:get_group(group_id=71818c8c50704a6ca8dfade2fa6e5a7d)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:06.759214 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4edf84c3-3cbf-484e-b190-16d46483902b None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:06.767036 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 188/378] 10.222.0.25 () {56 vars in 1180 bytes} [Fri Oct 28 09:31:06 2022] GET /identity/v3/groups/71818c8c50704a6ca8dfade2fa6e5a7d => generated 226 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:06.775560 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4edf84c3-3cbf-484e-b190-16d46483902b None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:06.790439 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c3349f-b7c7-45b2-8a07-747c79423dac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:06.791419 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c3349f-b7c7-45b2-8a07-747c79423dac None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:06.791923 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c3349f-b7c7-45b2-8a07-747c79423dac None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:06.792394 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c3349f-b7c7-45b2-8a07-747c79423dac None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:06.793387 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03c3349f-b7c7-45b2-8a07-747c79423dac None admin] RBAC: Adding query filter params (group.id=1c414fe7bf464383a9eee4581f8f3c2f, role.id=e13d798a98ea4c3dbb87a236ec683d32, scope.project.id=ac54d153685c4b2288a7ed215d0e24d2) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:06.793900 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03c3349f-b7c7-45b2-8a07-747c79423dac None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:06.796466 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03c3349f-b7c7-45b2-8a07-747c79423dac None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:06.803409 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 191/379] 10.222.0.25 () {56 vars in 1376 bytes} [Fri Oct 28 09:31:06 2022] GET /identity/v3/role_assignments?group.id=1c414fe7bf464383a9eee4581f8f3c2f&scope.project.id=ac54d153685c4b2288a7ed215d0e24d2&role.id=e13d798a98ea4c3dbb87a236ec683d32 => generated 260 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:06.930809 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:06.952539 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4edf84c3-3cbf-484e-b190-16d46483902b None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:06.998024 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c40079-eb8e-4f79-96c9-598c2f6912ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:06.998684 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c40079-eb8e-4f79-96c9-598c2f6912ef None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:06.998963 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c40079-eb8e-4f79-96c9-598c2f6912ef None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:06.999240 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c40079-eb8e-4f79-96c9-598c2f6912ef None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:06.999978 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c40079-eb8e-4f79-96c9-598c2f6912ef None admin] RBAC: Adding query filter params (group.id=71818c8c50704a6ca8dfade2fa6e5a7d, role.id=d93b4c94b590457199c3d301f947b0fb, scope.project.id=5960af42464f4e83b60ec52264d04b70) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:07.000284 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c40079-eb8e-4f79-96c9-598c2f6912ef None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:07.001713 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c40079-eb8e-4f79-96c9-598c2f6912ef None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:07.008277 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 189/380] 10.222.0.25 () {56 vars in 1376 bytes} [Fri Oct 28 09:31:06 2022] GET /identity/v3/role_assignments?group.id=71818c8c50704a6ca8dfade2fa6e5a7d&scope.project.id=5960af42464f4e83b60ec52264d04b70&role.id=d93b4c94b590457199c3d301f947b0fb => generated 260 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:31:07.656922 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bbbe88d-7c56-49bc-81d3-f153605f5d07 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:07.657702 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bbbe88d-7c56-49bc-81d3-f153605f5d07 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:07.657702 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bbbe88d-7c56-49bc-81d3-f153605f5d07 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:07.659369 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 192/381] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:07.664616 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af3bd446-6115-4f30-a158-76e4d901e6ae None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:07.664900 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af3bd446-6115-4f30-a158-76e4d901e6ae None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:07.665174 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af3bd446-6115-4f30-a158-76e4d901e6ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:07.687857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-af3bd446-6115-4f30-a158-76e4d901e6ae None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:07.689472 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-af3bd446-6115-4f30-a158-76e4d901e6ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:07.731016 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498ee1d7-cea4-4d1f-aa86-9edacf69b7db None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:07.731318 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498ee1d7-cea4-4d1f-aa86-9edacf69b7db None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:07.731566 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498ee1d7-cea4-4d1f-aa86-9edacf69b7db None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:07.733497 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 193/382] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 09:31:07.738678 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5815f18e-6c42-46b5-880a-047fd14a788f None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:07.738966 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5815f18e-6c42-46b5-880a-047fd14a788f None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:07.739243 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5815f18e-6c42-46b5-880a-047fd14a788f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:07.776063 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-5815f18e-6c42-46b5-880a-047fd14a788f None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:07.777276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-5815f18e-6c42-46b5-880a-047fd14a788f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:07.790208 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 190/383] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:31:07.885238 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 194/384] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 09:31:08.172734 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af3bd446-6115-4f30-a158-76e4d901e6ae None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:08.210719 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5815f18e-6c42-46b5-880a-047fd14a788f None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:08.213990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:08.255831 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:08.315231 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af3bd446-6115-4f30-a158-76e4d901e6ae None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:08.358544 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5815f18e-6c42-46b5-880a-047fd14a788f None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:08.381887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90e699d-69f1-452c-96c6-140db855ee54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:08.383117 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90e699d-69f1-452c-96c6-140db855ee54 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:08.383709 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90e699d-69f1-452c-96c6-140db855ee54 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:08.384198 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90e699d-69f1-452c-96c6-140db855ee54 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:08.402351 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90e699d-69f1-452c-96c6-140db855ee54 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:31:08.406719 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bad739b9-75c0-4219-a92b-7beacba91f5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:08.407629 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad739b9-75c0-4219-a92b-7beacba91f5a None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:08.407892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad739b9-75c0-4219-a92b-7beacba91f5a None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:08.408164 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad739b9-75c0-4219-a92b-7beacba91f5a None admin] PATH_INFO: `/v3/roles/member` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:08.409099 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-b90e699d-69f1-452c-96c6-140db855ee54 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 09:31:08.409099 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:08.409099 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:08.409099 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:08.409099 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:08.409099 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:08.409099 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:08.409099 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:08.409099 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:08.409099 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:08.409099 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:08.409099 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:08.409099 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:31:08.409099 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:31:08.409099 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:31:08.409099 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:31:08.409957 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:31:08.409957 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:31:08.409957 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:31:08.409957 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:31:08.409957 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:08.409957 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:08.409957 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:08.409957 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:08.409957 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:08.409957 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:08.409957 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:08.409957 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:08.409957 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:08.409957 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:08.409957 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:08.409957 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:08.410792 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:08.410792 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:08.410792 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:08.410792 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:08.410792 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:31:08.410792 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:31:08.410792 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:31:08.410792 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:31:08.410792 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:31:08.410792 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:31:08.410792 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 09:31:08.410792 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:08.413028 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 191/385] 10.222.0.25 () {56 vars in 1114 bytes} [Fri Oct 28 09:31:08 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:08.415062 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d31358-a8d5-4224-8a48-90bf3f10da75 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:08.415551 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d31358-a8d5-4224-8a48-90bf3f10da75 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:08.416025 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d31358-a8d5-4224-8a48-90bf3f10da75 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:08.417978 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 192/386] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:08.418937 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d31358-a8d5-4224-8a48-90bf3f10da75 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:08.430731 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bad739b9-75c0-4219-a92b-7beacba91f5a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:31:08.437752 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-bad739b9-75c0-4219-a92b-7beacba91f5a None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 09:31:08.437752 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:08.437752 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:08.437752 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:08.437752 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:08.437752 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:08.437752 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:08.437752 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:08.437752 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:08.437752 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:08.437752 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:08.437752 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:08.437752 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:31:08.437752 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:31:08.437752 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:31:08.437752 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:31:08.438880 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:31:08.438880 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:31:08.438880 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:31:08.438880 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:31:08.438880 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:08.438880 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:08.438880 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:08.438880 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:08.438880 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:08.438880 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:08.438880 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:08.438880 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:08.438880 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:08.438880 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:08.438880 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:08.438880 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:08.439676 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:08.439676 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:08.439676 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:08.439676 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:08.439676 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:31:08.439676 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:31:08.439676 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:31:08.439676 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:31:08.439676 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:31:08.439676 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:31:08.439676 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 09:31:08.439676 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:08.439676 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 195/387] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:31:08 2022] GET /identity/v3/roles/member => generated 84 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:31:08.440379 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622e96e5-4c54-49ee-b768-02a02a51800f None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:08.441057 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622e96e5-4c54-49ee-b768-02a02a51800f None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:08.441057 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-622e96e5-4c54-49ee-b768-02a02a51800f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:08.467422 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-622e96e5-4c54-49ee-b768-02a02a51800f None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:08.468634 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-622e96e5-4c54-49ee-b768-02a02a51800f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:08.538328 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:08.551546 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 196/388] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:31:08.552497 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622e96e5-4c54-49ee-b768-02a02a51800f None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:08.583552 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d31358-a8d5-4224-8a48-90bf3f10da75 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:08.645501 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:08.663099 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8178e30-a858-4b6a-8b61-19b592e3a65d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:08.663769 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8178e30-a858-4b6a-8b61-19b592e3a65d None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:08.664050 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8178e30-a858-4b6a-8b61-19b592e3a65d None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:08.664322 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8178e30-a858-4b6a-8b61-19b592e3a65d None admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:08.664987 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8178e30-a858-4b6a-8b61-19b592e3a65d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:08.665289 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8178e30-a858-4b6a-8b61-19b592e3a65d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:08.674747 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8178e30-a858-4b6a-8b61-19b592e3a65d None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:08.687581 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 193/389] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:31:08 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:08.696865 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8178e30-a858-4b6a-8b61-19b592e3a65d None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:08.700624 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-622e96e5-4c54-49ee-b768-02a02a51800f None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:08.744418 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e762002b-fcca-4489-9590-d2c467378b01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:08.745265 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e762002b-fcca-4489-9590-d2c467378b01 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:08.745549 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e762002b-fcca-4489-9590-d2c467378b01 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:08.745825 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e762002b-fcca-4489-9590-d2c467378b01 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:08.746524 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e762002b-fcca-4489-9590-d2c467378b01 None admin] RBAC: Adding query filter params (name=member) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:08.746837 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e762002b-fcca-4489-9590-d2c467378b01 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:08.753346 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e762002b-fcca-4489-9590-d2c467378b01 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:08.772129 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 197/390] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Oct 28 09:31:08 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:08.775360 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e762002b-fcca-4489-9590-d2c467378b01 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:08.854483 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:08.912348 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8178e30-a858-4b6a-8b61-19b592e3a65d None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:08.913331 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e762002b-fcca-4489-9590-d2c467378b01 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:08.952502 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98181a12-70f4-403f-9ae9-6021be1eeb99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:08.953988 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98181a12-70f4-403f-9ae9-6021be1eeb99 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:08.954497 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98181a12-70f4-403f-9ae9-6021be1eeb99 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:08.954950 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98181a12-70f4-403f-9ae9-6021be1eeb99 None admin] PATH_INFO: `/v3/groups/71818c8c50704a6ca8dfade2fa6e5a7d` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:08.956294 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98181a12-70f4-403f-9ae9-6021be1eeb99 None admin] RBAC: Authorizing `identity:get_group(group_id=71818c8c50704a6ca8dfade2fa6e5a7d)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:08.957818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1955329-5766-44af-a301-c71139c37559 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:08.958101 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98181a12-70f4-403f-9ae9-6021be1eeb99 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:08.958628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1955329-5766-44af-a301-c71139c37559 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:08.958991 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1955329-5766-44af-a301-c71139c37559 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:08.959343 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1955329-5766-44af-a301-c71139c37559 None admin] PATH_INFO: `/v3/groups/71818c8c50704a6ca8dfade2fa6e5a7d` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:08.960628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1955329-5766-44af-a301-c71139c37559 None admin] RBAC: Authorizing `identity:get_group(group_id=71818c8c50704a6ca8dfade2fa6e5a7d)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:08.966435 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1955329-5766-44af-a301-c71139c37559 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:08.966435 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 194/392] 10.222.0.25 () {56 vars in 1180 bytes} [Fri Oct 28 09:31:08 2022] GET /identity/v3/groups/71818c8c50704a6ca8dfade2fa6e5a7d => generated 226 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:08.966435 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1955329-5766-44af-a301-c71139c37559 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:08.966840 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 198/392] 10.222.0.25 () {56 vars in 1180 bytes} [Fri Oct 28 09:31:08 2022] GET /identity/v3/groups/71818c8c50704a6ca8dfade2fa6e5a7d => generated 226 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:31:08.967872 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98181a12-70f4-403f-9ae9-6021be1eeb99 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:09.052067 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:09.098113 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98181a12-70f4-403f-9ae9-6021be1eeb99 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:09.109267 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1955329-5766-44af-a301-c71139c37559 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:09.126216 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:09.141425 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14906960-0691-4367-bfe5-8451089b1d07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:09.142423 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14906960-0691-4367-bfe5-8451089b1d07 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:09.142911 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14906960-0691-4367-bfe5-8451089b1d07 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:09.143645 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14906960-0691-4367-bfe5-8451089b1d07 None admin] PATH_INFO: `/v3/projects/5960af42464f4e83b60ec52264d04b70` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:09.145766 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14906960-0691-4367-bfe5-8451089b1d07 None admin] RBAC: Authorizing `identity:get_project(project_id=5960af42464f4e83b60ec52264d04b70)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:09.147646 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14906960-0691-4367-bfe5-8451089b1d07 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:09.154575 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1239dd28-4b7e-422b-a8c5-09cf0896a0d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:09.154575 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1239dd28-4b7e-422b-a8c5-09cf0896a0d0 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:09.154575 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1239dd28-4b7e-422b-a8c5-09cf0896a0d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:09.154575 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1239dd28-4b7e-422b-a8c5-09cf0896a0d0 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:09.155370 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 199/393] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:31:08 2022] GET /identity/v3/projects/5960af42464f4e83b60ec52264d04b70 => generated 302 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:09.157102 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14906960-0691-4367-bfe5-8451089b1d07 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:09.166603 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1239dd28-4b7e-422b-a8c5-09cf0896a0d0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:31:09.175268 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-1239dd28-4b7e-422b-a8c5-09cf0896a0d0 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 09:31:09.175268 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:09.175268 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:09.175268 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:09.175268 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:09.175268 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:09.175268 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:09.175268 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:09.175268 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:09.175268 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:09.175268 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:09.175268 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:09.175268 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:31:09.175268 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:31:09.175268 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:31:09.175268 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:31:09.176182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:31:09.176182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:31:09.176182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:31:09.176182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:31:09.176182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:09.176182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:09.176182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:09.176182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:09.176182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:09.176182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:09.176182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:09.176182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:09.176182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:09.176182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:09.176182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:09.176182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:09.177007 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:09.177007 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:09.177007 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:09.177007 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:09.177007 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:31:09.177007 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:31:09.177007 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:31:09.177007 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:31:09.177007 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:31:09.177007 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:31:09.177007 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 09:31:09.177007 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:09.181825 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 195/394] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:31:08 2022] GET /identity/v3/roles/member => generated 84 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:09.182848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1239dd28-4b7e-422b-a8c5-09cf0896a0d0 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:09.286173 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14906960-0691-4367-bfe5-8451089b1d07 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:09.326071 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1239dd28-4b7e-422b-a8c5-09cf0896a0d0 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:09.328089 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a158868a-f31f-4fad-bb10-2b6584e94073 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:09.328995 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a158868a-f31f-4fad-bb10-2b6584e94073 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:09.329487 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a158868a-f31f-4fad-bb10-2b6584e94073 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:09.329987 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a158868a-f31f-4fad-bb10-2b6584e94073 None admin] PATH_INFO: `/v3/projects/6e28083c55ff4290af640f011fd2466f` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:09.331512 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a158868a-f31f-4fad-bb10-2b6584e94073 None admin] RBAC: Authorizing `identity:get_project(project_id=6e28083c55ff4290af640f011fd2466f)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:09.333349 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a158868a-f31f-4fad-bb10-2b6584e94073 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:09.340915 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 200/395] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:31:09 2022] GET /identity/v3/projects/6e28083c55ff4290af640f011fd2466f => generated 298 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:09.342245 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a158868a-f31f-4fad-bb10-2b6584e94073 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:09.350532 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:09.374824 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d9dbbe-5c28-4a5a-9603-181f0c699051 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:09.376128 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d9dbbe-5c28-4a5a-9603-181f0c699051 None admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:09.377025 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d9dbbe-5c28-4a5a-9603-181f0c699051 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:09.377630 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d9dbbe-5c28-4a5a-9603-181f0c699051 None admin] PATH_INFO: `/v3/projects/5960af42464f4e83b60ec52264d04b70/groups/71818c8c50704a6ca8dfade2fa6e5a7d/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:09.380553 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71d9dbbe-5c28-4a5a-9603-181f0c699051 None admin] RBAC: Authorizing `identity:create_grant(project_id=5960af42464f4e83b60ec52264d04b70, group_id=71818c8c50704a6ca8dfade2fa6e5a7d, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:09.382586 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71d9dbbe-5c28-4a5a-9603-181f0c699051 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:09.401893 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 196/396] 10.222.0.25 () {58 vars in 1442 bytes} [Fri Oct 28 09:31:09 2022] PUT /identity/v3/projects/5960af42464f4e83b60ec52264d04b70/groups/71818c8c50704a6ca8dfade2fa6e5a7d/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:31:09.403036 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d9dbbe-5c28-4a5a-9603-181f0c699051 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:09.425287 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:09.479397 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a158868a-f31f-4fad-bb10-2b6584e94073 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:09.520069 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692bc309-64fd-48a6-b576-734b875ef99a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:09.521138 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692bc309-64fd-48a6-b576-734b875ef99a None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:09.521138 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692bc309-64fd-48a6-b576-734b875ef99a None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:09.521490 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692bc309-64fd-48a6-b576-734b875ef99a None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:09.522364 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-692bc309-64fd-48a6-b576-734b875ef99a None admin] RBAC: Adding query filter params (name=member) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:09.522803 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-692bc309-64fd-48a6-b576-734b875ef99a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:09.524556 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-692bc309-64fd-48a6-b576-734b875ef99a None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:09.525138 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d9dbbe-5c28-4a5a-9603-181f0c699051 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:09.534738 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 201/397] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Oct 28 09:31:09 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:09.536151 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ed7fc0-ac33-4906-8dad-4a8f714a64d3 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:09.536568 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ed7fc0-ac33-4906-8dad-4a8f714a64d3 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:09.536854 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ed7fc0-ac33-4906-8dad-4a8f714a64d3 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:09.539186 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 202/398] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:09.547301 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c048b1f-5ac1-4d5f-8365-d251b0c9a696 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:09.547738 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c048b1f-5ac1-4d5f-8365-d251b0c9a696 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:09.548022 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c048b1f-5ac1-4d5f-8365-d251b0c9a696 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:09.573209 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f02ece-b195-4d1f-94b1-1379dcb75adb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:09.573983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f02ece-b195-4d1f-94b1-1379dcb75adb None admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:09.574316 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f02ece-b195-4d1f-94b1-1379dcb75adb None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:09.574672 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f02ece-b195-4d1f-94b1-1379dcb75adb None admin] PATH_INFO: `/v3/projects/6e28083c55ff4290af640f011fd2466f/groups/71818c8c50704a6ca8dfade2fa6e5a7d/roles/d93b4c94b590457199c3d301f947b0fb` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:09.576889 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-3c048b1f-5ac1-4d5f-8365-d251b0c9a696 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:09.577044 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9f02ece-b195-4d1f-94b1-1379dcb75adb None admin] RBAC: Authorizing `identity:create_grant(project_id=6e28083c55ff4290af640f011fd2466f, group_id=71818c8c50704a6ca8dfade2fa6e5a7d, role_id=d93b4c94b590457199c3d301f947b0fb)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:09.578432 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-3c048b1f-5ac1-4d5f-8365-d251b0c9a696 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:09.578558 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9f02ece-b195-4d1f-94b1-1379dcb75adb None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:09.598912 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 197/399] 10.222.0.25 () {58 vars in 1442 bytes} [Fri Oct 28 09:31:09 2022] PUT /identity/v3/projects/6e28083c55ff4290af640f011fd2466f/groups/71818c8c50704a6ca8dfade2fa6e5a7d/roles/d93b4c94b590457199c3d301f947b0fb => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:31:09.599568 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f02ece-b195-4d1f-94b1-1379dcb75adb None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:09.668993 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:09.685560 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 203/400] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:31:09.755148 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f02ece-b195-4d1f-94b1-1379dcb75adb None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:09.795152 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d355a6e3-891e-4955-a65d-c174cd0079e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:09.795784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d355a6e3-891e-4955-a65d-c174cd0079e9 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:09.796072 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d355a6e3-891e-4955-a65d-c174cd0079e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:09.796357 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d355a6e3-891e-4955-a65d-c174cd0079e9 None admin] PATH_INFO: `/v3/groups/71818c8c50704a6ca8dfade2fa6e5a7d` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:09.797619 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d355a6e3-891e-4955-a65d-c174cd0079e9 None admin] RBAC: Authorizing `identity:get_group(group_id=71818c8c50704a6ca8dfade2fa6e5a7d)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:09.799132 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d355a6e3-891e-4955-a65d-c174cd0079e9 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:09.800755 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 198/401] 10.222.0.25 () {56 vars in 1180 bytes} [Fri Oct 28 09:31:09 2022] GET /identity/v3/groups/71818c8c50704a6ca8dfade2fa6e5a7d => generated 226 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:09.807683 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c048b1f-5ac1-4d5f-8365-d251b0c9a696 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:09.816287 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad54fa2b-603e-42f4-9d12-0a9999f9188d None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:09.816577 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad54fa2b-603e-42f4-9d12-0a9999f9188d None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:09.816856 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad54fa2b-603e-42f4-9d12-0a9999f9188d None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:09.818317 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 199/402] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 09:31:09.823773 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d434b83-8467-4e0c-9640-4f729ec64d2f None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:09.824337 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d434b83-8467-4e0c-9640-4f729ec64d2f None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:09.824809 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d434b83-8467-4e0c-9640-4f729ec64d2f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:09.844464 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-8d434b83-8467-4e0c-9640-4f729ec64d2f None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:09.846018 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-8d434b83-8467-4e0c-9640-4f729ec64d2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:09.864342 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:09.937314 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 200/403] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 09:31:09.938280 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c048b1f-5ac1-4d5f-8365-d251b0c9a696 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:09.993500 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bddf66a-1511-4167-a489-8e006b973fdf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:09.994454 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bddf66a-1511-4167-a489-8e006b973fdf None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:09.994933 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bddf66a-1511-4167-a489-8e006b973fdf None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:09.995387 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bddf66a-1511-4167-a489-8e006b973fdf None admin] PATH_INFO: `/v3/projects/6e28083c55ff4290af640f011fd2466f` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:09.996671 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bddf66a-1511-4167-a489-8e006b973fdf None admin] RBAC: Authorizing `identity:get_project(project_id=6e28083c55ff4290af640f011fd2466f)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:09.998264 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bddf66a-1511-4167-a489-8e006b973fdf None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:10.004439 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 204/404] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:31:09 2022] GET /identity/v3/projects/6e28083c55ff4290af640f011fd2466f => generated 298 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:10.010434 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d434b83-8467-4e0c-9640-4f729ec64d2f None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:10.043533 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bddf66a-1511-4167-a489-8e006b973fdf None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:10.080333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:10.216534 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d434b83-8467-4e0c-9640-4f729ec64d2f None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:10.222079 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bddf66a-1511-4167-a489-8e006b973fdf None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:10.260078 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:10.285034 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e7e7f3-e656-4011-8904-0518fd21d027 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:10.286088 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e7e7f3-e656-4011-8904-0518fd21d027 None admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:10.286616 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e7e7f3-e656-4011-8904-0518fd21d027 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:10.288156 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e7e7f3-e656-4011-8904-0518fd21d027 None admin] PATH_INFO: `/v3/projects/6e28083c55ff4290af640f011fd2466f/groups/71818c8c50704a6ca8dfade2fa6e5a7d/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:10.292846 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2e7e7f3-e656-4011-8904-0518fd21d027 None admin] RBAC: Authorizing `identity:create_grant(project_id=6e28083c55ff4290af640f011fd2466f, group_id=71818c8c50704a6ca8dfade2fa6e5a7d, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:10.296062 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2e7e7f3-e656-4011-8904-0518fd21d027 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:10.301354 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90da9c6f-12b0-474c-8810-62b1c9f867fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:10.302028 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90da9c6f-12b0-474c-8810-62b1c9f867fb None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:10.302523 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90da9c6f-12b0-474c-8810-62b1c9f867fb None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:10.302799 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90da9c6f-12b0-474c-8810-62b1c9f867fb None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:10.316561 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 201/405] 10.222.0.25 () {58 vars in 1442 bytes} [Fri Oct 28 09:31:10 2022] PUT /identity/v3/projects/6e28083c55ff4290af640f011fd2466f/groups/71818c8c50704a6ca8dfade2fa6e5a7d/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 307 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:31:10.320556 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e7e7f3-e656-4011-8904-0518fd21d027 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:10.324555 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90da9c6f-12b0-474c-8810-62b1c9f867fb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:31:10.330064 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-90da9c6f-12b0-474c-8810-62b1c9f867fb None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:31:10.330064 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:10.330064 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:10.330064 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:10.330064 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:10.330064 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:10.330064 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:10.330064 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:10.330064 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:10.330064 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:10.330064 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:10.330064 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:10.330064 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:31:10.330064 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:31:10.330064 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:31:10.330064 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:31:10.331062 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:31:10.331062 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:31:10.331062 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:31:10.331062 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:31:10.331062 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:10.331062 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:10.331062 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:10.331062 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:10.331062 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:10.331062 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:10.331062 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:10.331062 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:10.331062 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:10.331062 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:10.331062 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:10.331062 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:10.331868 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:10.331868 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:10.331868 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:10.331868 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:10.331868 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:31:10.331868 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:31:10.331868 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:31:10.331868 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:31:10.331868 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:31:10.331868 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:31:10.331868 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:31:10.331868 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:10.331868 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 205/406] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Oct 28 09:31:10 2022] GET /identity/v3/roles/admin => generated 83 bytes in 288 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:31:10.337271 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90da9c6f-12b0-474c-8810-62b1c9f867fb None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:10.514957 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90da9c6f-12b0-474c-8810-62b1c9f867fb None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:10.535120 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:10.548039 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e7e7f3-e656-4011-8904-0518fd21d027 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:10.567221 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7261c353-da4b-4b2f-881c-88ac5902571f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:10.568215 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7261c353-da4b-4b2f-881c-88ac5902571f None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:10.568501 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7261c353-da4b-4b2f-881c-88ac5902571f None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:10.568842 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7261c353-da4b-4b2f-881c-88ac5902571f None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:10.569857 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7261c353-da4b-4b2f-881c-88ac5902571f None admin] RBAC: Adding query filter params (name=admin) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:10.570185 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7261c353-da4b-4b2f-881c-88ac5902571f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:10.572345 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7261c353-da4b-4b2f-881c-88ac5902571f None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:10.588310 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 206/407] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:31:10 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:31:10.594079 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7261c353-da4b-4b2f-881c-88ac5902571f None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:10.604689 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee383c3-4aaa-4daa-a180-6e8e2ba85cab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:10.605584 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee383c3-4aaa-4daa-a180-6e8e2ba85cab None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:10.606006 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee383c3-4aaa-4daa-a180-6e8e2ba85cab None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:10.606442 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee383c3-4aaa-4daa-a180-6e8e2ba85cab None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:10.621554 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ee383c3-4aaa-4daa-a180-6e8e2ba85cab None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:31:10.630792 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-2ee383c3-4aaa-4daa-a180-6e8e2ba85cab None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 09:31:10.630792 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:10.630792 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:10.630792 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:10.630792 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:10.630792 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:10.630792 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:10.630792 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:10.630792 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:10.630792 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:10.630792 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:10.630792 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:10.630792 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:31:10.630792 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:31:10.630792 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:31:10.630792 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:31:10.631936 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:31:10.631936 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:31:10.631936 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:31:10.631936 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:31:10.631936 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:10.631936 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:10.631936 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:10.631936 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:10.631936 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:10.631936 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:10.631936 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:10.631936 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:10.631936 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:10.631936 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:10.631936 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:10.631936 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:10.632851 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:10.632851 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:10.632851 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:10.632851 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:10.632851 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:31:10.632851 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:31:10.632851 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:31:10.632851 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:31:10.632851 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:31:10.632851 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:31:10.632851 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 09:31:10.632851 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:10.632851 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 202/408] 10.222.0.25 () {56 vars in 1114 bytes} [Fri Oct 28 09:31:10 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 315 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:10.642419 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee383c3-4aaa-4daa-a180-6e8e2ba85cab None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:10.668611 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:10.760884 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee383c3-4aaa-4daa-a180-6e8e2ba85cab None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:10.799707 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4e5770-ec27-49a8-8d10-12eba3108630 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:10.801179 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4e5770-ec27-49a8-8d10-12eba3108630 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:10.802078 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4e5770-ec27-49a8-8d10-12eba3108630 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:10.802522 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4e5770-ec27-49a8-8d10-12eba3108630 None admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:10.803397 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc4e5770-ec27-49a8-8d10-12eba3108630 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:10.803843 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc4e5770-ec27-49a8-8d10-12eba3108630 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:10.805947 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc4e5770-ec27-49a8-8d10-12eba3108630 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:10.816473 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 203/409] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:31:10 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:31:10.818685 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7261c353-da4b-4b2f-881c-88ac5902571f None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:10.822625 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4e5770-ec27-49a8-8d10-12eba3108630 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:10.864452 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:10.903371 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecedb17a-7729-41b3-bd0b-cf304b0a6bfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:10.904347 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecedb17a-7729-41b3-bd0b-cf304b0a6bfc None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:10.904762 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecedb17a-7729-41b3-bd0b-cf304b0a6bfc None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:10.905172 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecedb17a-7729-41b3-bd0b-cf304b0a6bfc None admin] PATH_INFO: `/v3/groups/1c414fe7bf464383a9eee4581f8f3c2f` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:10.907076 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecedb17a-7729-41b3-bd0b-cf304b0a6bfc None admin] RBAC: Authorizing `identity:get_group(group_id=1c414fe7bf464383a9eee4581f8f3c2f)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:10.911757 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecedb17a-7729-41b3-bd0b-cf304b0a6bfc None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:10.916659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 207/410] 10.222.0.25 () {56 vars in 1180 bytes} [Fri Oct 28 09:31:10 2022] GET /identity/v3/groups/1c414fe7bf464383a9eee4581f8f3c2f => generated 229 bytes in 323 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:31:10.922643 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecedb17a-7729-41b3-bd0b-cf304b0a6bfc None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:10.994059 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4e5770-ec27-49a8-8d10-12eba3108630 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:11.034903 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8855f01-93a9-4df6-8e7d-7cabd48ce8d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:11.035919 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8855f01-93a9-4df6-8e7d-7cabd48ce8d7 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:11.036336 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8855f01-93a9-4df6-8e7d-7cabd48ce8d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:11.036757 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8855f01-93a9-4df6-8e7d-7cabd48ce8d7 None admin] PATH_INFO: `/v3/groups/71818c8c50704a6ca8dfade2fa6e5a7d` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:11.038215 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8855f01-93a9-4df6-8e7d-7cabd48ce8d7 None admin] RBAC: Authorizing `identity:get_group(group_id=71818c8c50704a6ca8dfade2fa6e5a7d)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:11.040366 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8855f01-93a9-4df6-8e7d-7cabd48ce8d7 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:11.042631 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 204/411] 10.222.0.25 () {56 vars in 1180 bytes} [Fri Oct 28 09:31:10 2022] GET /identity/v3/groups/71818c8c50704a6ca8dfade2fa6e5a7d => generated 226 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:31:11.048261 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8855f01-93a9-4df6-8e7d-7cabd48ce8d7 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:11.138737 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecedb17a-7729-41b3-bd0b-cf304b0a6bfc None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:11.159355 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:11.215974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:11.227326 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd74daf-81d0-417d-b3b7-236727b3a984 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:11.228144 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd74daf-81d0-417d-b3b7-236727b3a984 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:11.228574 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd74daf-81d0-417d-b3b7-236727b3a984 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:11.228882 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd74daf-81d0-417d-b3b7-236727b3a984 None admin] PATH_INFO: `/v3/projects/ac54d153685c4b2288a7ed215d0e24d2` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:11.230186 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bd74daf-81d0-417d-b3b7-236727b3a984 None admin] RBAC: Authorizing `identity:get_project(project_id=ac54d153685c4b2288a7ed215d0e24d2)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:11.231728 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bd74daf-81d0-417d-b3b7-236727b3a984 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:11.233809 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 208/412] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:31:10 2022] GET /identity/v3/projects/ac54d153685c4b2288a7ed215d0e24d2 => generated 344 bytes in 312 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:31:11.234873 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8855f01-93a9-4df6-8e7d-7cabd48ce8d7 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:11.239216 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd74daf-81d0-417d-b3b7-236727b3a984 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:11.274415 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e592186-d90b-4087-bbec-e1bab17cd7bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:11.275142 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e592186-d90b-4087-bbec-e1bab17cd7bb None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:11.275482 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e592186-d90b-4087-bbec-e1bab17cd7bb None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:11.275759 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e592186-d90b-4087-bbec-e1bab17cd7bb None admin] PATH_INFO: `/v3/projects/5960af42464f4e83b60ec52264d04b70` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:11.276918 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e592186-d90b-4087-bbec-e1bab17cd7bb None admin] RBAC: Authorizing `identity:get_project(project_id=5960af42464f4e83b60ec52264d04b70)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:11.278516 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e592186-d90b-4087-bbec-e1bab17cd7bb None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:11.280456 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 205/413] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:31:11 2022] GET /identity/v3/projects/5960af42464f4e83b60ec52264d04b70 => generated 302 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:11.291872 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e592186-d90b-4087-bbec-e1bab17cd7bb None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:11.377051 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd74daf-81d0-417d-b3b7-236727b3a984 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:11.444277 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ba474f-fe4c-49ae-8b52-7f84710be3d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:11.444958 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ba474f-fe4c-49ae-8b52-7f84710be3d8 None admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:11.445241 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ba474f-fe4c-49ae-8b52-7f84710be3d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:11.445515 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ba474f-fe4c-49ae-8b52-7f84710be3d8 None admin] PATH_INFO: `/v3/projects/ac54d153685c4b2288a7ed215d0e24d2/groups/1c414fe7bf464383a9eee4581f8f3c2f/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:11.448674 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1ba474f-fe4c-49ae-8b52-7f84710be3d8 None admin] RBAC: Authorizing `identity:create_grant(project_id=ac54d153685c4b2288a7ed215d0e24d2, group_id=1c414fe7bf464383a9eee4581f8f3c2f, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:11.450097 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1ba474f-fe4c-49ae-8b52-7f84710be3d8 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:11.473829 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 209/414] 10.222.0.25 () {58 vars in 1442 bytes} [Fri Oct 28 09:31:11 2022] PUT /identity/v3/projects/ac54d153685c4b2288a7ed215d0e24d2/groups/1c414fe7bf464383a9eee4581f8f3c2f/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:31:11.520052 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e592186-d90b-4087-bbec-e1bab17cd7bb None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:11.581750 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-867d7e13-ac4e-4183-8d10-c4bc9209fe53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:11.582489 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867d7e13-ac4e-4183-8d10-c4bc9209fe53 None admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:11.582822 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867d7e13-ac4e-4183-8d10-c4bc9209fe53 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:11.583106 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867d7e13-ac4e-4183-8d10-c4bc9209fe53 None admin] PATH_INFO: `/v3/projects/5960af42464f4e83b60ec52264d04b70/groups/71818c8c50704a6ca8dfade2fa6e5a7d/roles/d93b4c94b590457199c3d301f947b0fb` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:11.586075 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-867d7e13-ac4e-4183-8d10-c4bc9209fe53 None admin] RBAC: Authorizing `identity:create_grant(project_id=5960af42464f4e83b60ec52264d04b70, group_id=71818c8c50704a6ca8dfade2fa6e5a7d, role_id=d93b4c94b590457199c3d301f947b0fb)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:11.587612 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-867d7e13-ac4e-4183-8d10-c4bc9209fe53 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:11.608565 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 206/415] 10.222.0.25 () {58 vars in 1442 bytes} [Fri Oct 28 09:31:11 2022] PUT /identity/v3/projects/5960af42464f4e83b60ec52264d04b70/groups/71818c8c50704a6ca8dfade2fa6e5a7d/roles/d93b4c94b590457199c3d301f947b0fb => generated 0 bytes in 317 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:31:11.973148 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40f3c56-54d5-47d2-8c04-e07460e48bbf None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:11.974174 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40f3c56-54d5-47d2-8c04-e07460e48bbf None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:11.974684 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40f3c56-54d5-47d2-8c04-e07460e48bbf None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:11.977060 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 210/416] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:11 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:11.983146 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29435797-3748-45c2-a2ce-6e64c1eeab7f None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:11.983727 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29435797-3748-45c2-a2ce-6e64c1eeab7f None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:11.984234 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29435797-3748-45c2-a2ce-6e64c1eeab7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:12.004690 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-29435797-3748-45c2-a2ce-6e64c1eeab7f None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:12.006687 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-29435797-3748-45c2-a2ce-6e64c1eeab7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:12.024076 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:12.105095 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 207/417] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 09:31:12.410104 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca92ff72-263b-4ba1-8a64-7db8a5d51359 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:12.410828 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca92ff72-263b-4ba1-8a64-7db8a5d51359 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:12.411341 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca92ff72-263b-4ba1-8a64-7db8a5d51359 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:12.413213 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 211/418] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:12.421549 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e65b52-e8b5-430a-83e2-c85013fe01f6 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:12.422148 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e65b52-e8b5-430a-83e2-c85013fe01f6 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:12.422735 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e65b52-e8b5-430a-83e2-c85013fe01f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:12.447472 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-b9e65b52-e8b5-430a-83e2-c85013fe01f6 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:12.448690 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-b9e65b52-e8b5-430a-83e2-c85013fe01f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:12.494029 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca92ff72-263b-4ba1-8a64-7db8a5d51359 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:12.544499 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:12.553045 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 208/419] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:31:12.621688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca92ff72-263b-4ba1-8a64-7db8a5d51359 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:12.659341 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a232632d-c732-46c4-ad91-50b7263a45e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:12.660002 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a232632d-c732-46c4-ad91-50b7263a45e5 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:12.660342 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a232632d-c732-46c4-ad91-50b7263a45e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:12.660569 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a232632d-c732-46c4-ad91-50b7263a45e5 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:12.673380 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a232632d-c732-46c4-ad91-50b7263a45e5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:31:12.678330 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-a232632d-c732-46c4-ad91-50b7263a45e5 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 09:31:12.678330 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:12.678330 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:12.678330 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:12.678330 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:12.678330 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:12.678330 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:12.678330 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:12.678330 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:12.678330 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:12.678330 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:12.678330 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:12.678330 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:31:12.678330 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:31:12.678330 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:31:12.678330 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:31:12.679276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:31:12.679276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:31:12.679276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:31:12.679276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:31:12.679276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:12.679276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:12.679276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:12.679276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:12.679276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:12.679276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:12.679276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:12.679276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:12.679276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:12.679276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:12.679276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:12.679276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:12.680059 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:12.680059 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:12.680059 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:12.680059 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:12.680059 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:31:12.680059 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:31:12.680059 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:31:12.680059 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:31:12.680059 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:31:12.680059 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:31:12.680059 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 09:31:12.680059 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:12.684378 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 212/420] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:31:12 2022] GET /identity/v3/roles/member => generated 84 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:12.685829 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9e65b52-e8b5-430a-83e2-c85013fe01f6 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:12.796754 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:12.825823 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9e65b52-e8b5-430a-83e2-c85013fe01f6 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:12.858914 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a232632d-c732-46c4-ad91-50b7263a45e5 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:12.882417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9f5a2e-7f5b-4123-8e70-d6f902407a38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:12.882417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9f5a2e-7f5b-4123-8e70-d6f902407a38 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:12.882417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9f5a2e-7f5b-4123-8e70-d6f902407a38 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:12.882417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9f5a2e-7f5b-4123-8e70-d6f902407a38 None admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:12.882819 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9f5a2e-7f5b-4123-8e70-d6f902407a38 None admin] RBAC: Adding query filter params (name=member) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:12.882819 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9f5a2e-7f5b-4123-8e70-d6f902407a38 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:12.882819 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9f5a2e-7f5b-4123-8e70-d6f902407a38 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:12.891369 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 209/421] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Oct 28 09:31:12 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:12.893857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9f5a2e-7f5b-4123-8e70-d6f902407a38 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:13.039244 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:13.050270 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a232632d-c732-46c4-ad91-50b7263a45e5 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:13.058412 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9f5a2e-7f5b-4123-8e70-d6f902407a38 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:13.096990 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83228399-a2a5-47b1-9926-b3bfc582ab10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:13.097851 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83228399-a2a5-47b1-9926-b3bfc582ab10 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:13.098876 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83228399-a2a5-47b1-9926-b3bfc582ab10 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:13.099307 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83228399-a2a5-47b1-9926-b3bfc582ab10 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:13.124268 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:13.141828 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8062b40-a56a-4675-a08a-16d4fa50e3b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:13.142762 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8062b40-a56a-4675-a08a-16d4fa50e3b8 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:13.143274 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8062b40-a56a-4675-a08a-16d4fa50e3b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:13.143748 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8062b40-a56a-4675-a08a-16d4fa50e3b8 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:13.155252 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8062b40-a56a-4675-a08a-16d4fa50e3b8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:31:13.160678 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-a8062b40-a56a-4675-a08a-16d4fa50e3b8 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 09:31:13.160678 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:13.160678 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:13.160678 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:13.160678 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:13.160678 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:13.160678 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:13.160678 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:13.160678 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:13.160678 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:13.160678 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:13.160678 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:13.160678 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:31:13.160678 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:31:13.160678 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:31:13.160678 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:31:13.161565 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:31:13.161565 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:31:13.161565 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:31:13.161565 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:31:13.161565 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:13.161565 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:13.161565 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:13.161565 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:13.161565 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:13.161565 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:13.161565 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:13.161565 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:13.161565 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:13.161565 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:13.161565 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:13.161565 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:13.162369 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:13.162369 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:13.162369 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:13.162369 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:13.162369 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:31:13.162369 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:31:13.162369 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:31:13.162369 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:31:13.162369 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:31:13.162369 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:31:13.162369 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 09:31:13.162369 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:13.165128 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 213/422] 10.222.0.25 () {56 vars in 1114 bytes} [Fri Oct 28 09:31:12 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 307 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:31:13.166667 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9ec18d-2393-48cc-855f-b040443c7d00 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:13.167158 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9ec18d-2393-48cc-855f-b040443c7d00 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:13.167648 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b9ec18d-2393-48cc-855f-b040443c7d00 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:13.169651 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 214/423] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:13.170584 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b9ec18d-2393-48cc-855f-b040443c7d00 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:13.187052 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83228399-a2a5-47b1-9926-b3bfc582ab10 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:13.189299 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83228399-a2a5-47b1-9926-b3bfc582ab10 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:13.295502 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b9ec18d-2393-48cc-855f-b040443c7d00 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:13.318001 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 210/424] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:31:12 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 420 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 28 09:31:13.318001 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0ec88a-b024-4d41-8bbe-a57e06a5fe23 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:13.318001 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0ec88a-b024-4d41-8bbe-a57e06a5fe23 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:13.318001 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0ec88a-b024-4d41-8bbe-a57e06a5fe23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:13.339250 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-8f0ec88a-b024-4d41-8bbe-a57e06a5fe23 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:13.340754 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-8f0ec88a-b024-4d41-8bbe-a57e06a5fe23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:13.344811 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1c8a8b-c6d4-4c2f-8905-cdecdac63e99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:13.344811 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1c8a8b-c6d4-4c2f-8905-cdecdac63e99 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:13.345076 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1c8a8b-c6d4-4c2f-8905-cdecdac63e99 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:13.345492 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f1c8a8b-c6d4-4c2f-8905-cdecdac63e99 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:13.346222 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f1c8a8b-c6d4-4c2f-8905-cdecdac63e99 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:13.346625 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f1c8a8b-c6d4-4c2f-8905-cdecdac63e99 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:13.348045 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f1c8a8b-c6d4-4c2f-8905-cdecdac63e99 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:13.357863 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 215/425] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:31:13 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:13.363111 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1c8a8b-c6d4-4c2f-8905-cdecdac63e99 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:13.375847 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:13.386668 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:13.417561 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 211/426] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:31:13.419184 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0ec88a-b024-4d41-8bbe-a57e06a5fe23 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:13.515676 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f1c8a8b-c6d4-4c2f-8905-cdecdac63e99 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:13.545782 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0ec88a-b024-4d41-8bbe-a57e06a5fe23 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:13.566153 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84351d8c-f46d-4d15-b6ab-f3c31375a665 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:13.566153 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84351d8c-f46d-4d15-b6ab-f3c31375a665 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:13.566153 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84351d8c-f46d-4d15-b6ab-f3c31375a665 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:13.566153 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84351d8c-f46d-4d15-b6ab-f3c31375a665 None admin] PATH_INFO: `/v3/projects/5960af42464f4e83b60ec52264d04b70` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:13.566583 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84351d8c-f46d-4d15-b6ab-f3c31375a665 None admin] RBAC: Authorizing `identity:get_project(project_id=5960af42464f4e83b60ec52264d04b70)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:13.566583 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84351d8c-f46d-4d15-b6ab-f3c31375a665 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:13.567262 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 216/427] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:31:13 2022] GET /identity/v3/projects/5960af42464f4e83b60ec52264d04b70 => generated 302 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:13.577867 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84351d8c-f46d-4d15-b6ab-f3c31375a665 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:13.633518 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd642d91-5160-4958-bf72-81b16eb62b57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:13.634140 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd642d91-5160-4958-bf72-81b16eb62b57 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:13.634439 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd642d91-5160-4958-bf72-81b16eb62b57 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:13.634720 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd642d91-5160-4958-bf72-81b16eb62b57 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:13.673175 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:13.691718 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84351d8c-f46d-4d15-b6ab-f3c31375a665 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:13.701653 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:13.732619 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f2eea3d-922e-41bf-b1d4-c54731788a0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:13.733645 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2eea3d-922e-41bf-b1d4-c54731788a0d None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:13.734166 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2eea3d-922e-41bf-b1d4-c54731788a0d None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:13.734655 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2eea3d-922e-41bf-b1d4-c54731788a0d None admin] PATH_INFO: `/v3/groups/71818c8c50704a6ca8dfade2fa6e5a7d` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:13.735982 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f2eea3d-922e-41bf-b1d4-c54731788a0d None admin] RBAC: Authorizing `identity:get_group(group_id=71818c8c50704a6ca8dfade2fa6e5a7d)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:13.737713 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f2eea3d-922e-41bf-b1d4-c54731788a0d None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:13.739887 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 217/428] 10.222.0.25 () {56 vars in 1180 bytes} [Fri Oct 28 09:31:13 2022] GET /identity/v3/groups/71818c8c50704a6ca8dfade2fa6e5a7d => generated 226 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:13.746207 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f2eea3d-922e-41bf-b1d4-c54731788a0d None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:13.754123 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd642d91-5160-4958-bf72-81b16eb62b57 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:13.755701 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd642d91-5160-4958-bf72-81b16eb62b57 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:13.888369 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f2eea3d-922e-41bf-b1d4-c54731788a0d None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:13.898157 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 212/429] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:31:13 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 480 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 09:31:13.904087 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd642d91-5160-4958-bf72-81b16eb62b57 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:13.941950 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60699037-fae4-46fb-8384-0150234a7a2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:13.943113 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60699037-fae4-46fb-8384-0150234a7a2e None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:13.943709 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60699037-fae4-46fb-8384-0150234a7a2e None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:13.945284 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60699037-fae4-46fb-8384-0150234a7a2e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:13.947138 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60699037-fae4-46fb-8384-0150234a7a2e None admin] RBAC: Adding query filter params (group.id=71818c8c50704a6ca8dfade2fa6e5a7d, role.id=8943f289c9fe4feba2695b9c90cf3d51, scope.project.id=5960af42464f4e83b60ec52264d04b70) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:13.947710 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60699037-fae4-46fb-8384-0150234a7a2e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:13.949758 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60699037-fae4-46fb-8384-0150234a7a2e None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:13.958369 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 218/430] 10.222.0.25 () {56 vars in 1376 bytes} [Fri Oct 28 09:31:13 2022] GET /identity/v3/role_assignments?group.id=71818c8c50704a6ca8dfade2fa6e5a7d&scope.project.id=5960af42464f4e83b60ec52264d04b70&role.id=8943f289c9fe4feba2695b9c90cf3d51 => generated 611 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:13.966423 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60699037-fae4-46fb-8384-0150234a7a2e None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:13.996095 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:14.008091 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:14.063936 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd642d91-5160-4958-bf72-81b16eb62b57 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:14.121515 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b49369c-b78d-4eb0-84dc-70bb02c182f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:14.122199 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b49369c-b78d-4eb0-84dc-70bb02c182f0 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:14.122518 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b49369c-b78d-4eb0-84dc-70bb02c182f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:14.122802 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b49369c-b78d-4eb0-84dc-70bb02c182f0 None admin] PATH_INFO: `/v3/projects/6e28083c55ff4290af640f011fd2466f` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:14.126425 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b49369c-b78d-4eb0-84dc-70bb02c182f0 None admin] RBAC: Authorizing `identity:get_project(project_id=6e28083c55ff4290af640f011fd2466f)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:14.126425 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b49369c-b78d-4eb0-84dc-70bb02c182f0 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:14.133385 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 213/431] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:31:13 2022] GET /identity/v3/projects/6e28083c55ff4290af640f011fd2466f => generated 298 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:14.136500 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b49369c-b78d-4eb0-84dc-70bb02c182f0 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:14.207516 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60699037-fae4-46fb-8384-0150234a7a2e None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:14.269286 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7f89c84-b3b3-4e00-852c-dd8898f70875 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:14.269989 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f89c84-b3b3-4e00-852c-dd8898f70875 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:14.270277 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f89c84-b3b3-4e00-852c-dd8898f70875 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:14.270777 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f89c84-b3b3-4e00-852c-dd8898f70875 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:14.285369 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7f89c84-b3b3-4e00-852c-dd8898f70875 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:31:14.292541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-f7f89c84-b3b3-4e00-852c-dd8898f70875 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Oct 28 09:31:14.292541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:14.292541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:14.292541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:14.292541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:14.292541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:14.292541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:14.292541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:14.292541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:14.292541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:14.292541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:14.292541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:14.292541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:31:14.292541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:31:14.292541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:31:14.292541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:31:14.293449 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:31:14.293449 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:31:14.293449 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:31:14.293449 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:31:14.293449 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:14.293449 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:14.293449 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:14.293449 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:14.293449 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:14.293449 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:14.293449 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:14.293449 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:14.293449 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:14.293449 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:14.293449 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:14.293449 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:14.294255 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:14.294255 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:14.294255 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:14.294255 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:14.294255 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:31:14.294255 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:31:14.294255 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:31:14.294255 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:31:14.294255 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:31:14.294255 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:31:14.294255 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Oct 28 09:31:14.294255 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:14.295525 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 219/432] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:31:13 2022] GET /identity/v3/roles/member => generated 84 bytes in 336 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:14.296480 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06b1bb5-246e-4578-8a0e-4989e758617f None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:14.296771 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06b1bb5-246e-4578-8a0e-4989e758617f None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:14.297115 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06b1bb5-246e-4578-8a0e-4989e758617f None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:14.300316 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 220/433] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:14.301630 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06b1bb5-246e-4578-8a0e-4989e758617f None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:14.308432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b49369c-b78d-4eb0-84dc-70bb02c182f0 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:14.350014 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e2aac7-2642-46a2-b375-b0f63686950f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:14.350731 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e2aac7-2642-46a2-b375-b0f63686950f None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:14.351082 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e2aac7-2642-46a2-b375-b0f63686950f None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:14.351399 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e2aac7-2642-46a2-b375-b0f63686950f None admin] PATH_INFO: `/v3/groups/71818c8c50704a6ca8dfade2fa6e5a7d` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:14.352616 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74e2aac7-2642-46a2-b375-b0f63686950f None admin] RBAC: Authorizing `identity:get_group(group_id=71818c8c50704a6ca8dfade2fa6e5a7d)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:14.354118 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74e2aac7-2642-46a2-b375-b0f63686950f None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:14.355780 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 214/434] 10.222.0.25 () {56 vars in 1180 bytes} [Fri Oct 28 09:31:14 2022] GET /identity/v3/groups/71818c8c50704a6ca8dfade2fa6e5a7d => generated 226 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:14.356944 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c63639-fc60-4dc1-8414-01ff50a031dd None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:14.357230 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c63639-fc60-4dc1-8414-01ff50a031dd None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:14.357505 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c63639-fc60-4dc1-8414-01ff50a031dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:14.363642 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:14.376823 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-a2c63639-fc60-4dc1-8414-01ff50a031dd None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:14.378065 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-a2c63639-fc60-4dc1-8414-01ff50a031dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:14.402752 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:14.425373 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06b1bb5-246e-4578-8a0e-4989e758617f None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:14.445393 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 215/435] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:31:14.447195 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b145ac61-b2ae-42fb-ae20-76d546bbbdb2 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:14.447471 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b145ac61-b2ae-42fb-ae20-76d546bbbdb2 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:14.447762 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b145ac61-b2ae-42fb-ae20-76d546bbbdb2 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:14.450951 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 216/436] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:14.450951 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b145ac61-b2ae-42fb-ae20-76d546bbbdb2 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:14.466377 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7e4a4c7-ef99-479f-91b0-97a25df06c9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:14.466992 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e4a4c7-ef99-479f-91b0-97a25df06c9a None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:14.467332 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e4a4c7-ef99-479f-91b0-97a25df06c9a None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:14.467611 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e4a4c7-ef99-479f-91b0-97a25df06c9a None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:14.468274 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7e4a4c7-ef99-479f-91b0-97a25df06c9a None admin] RBAC: Adding query filter params (name=member) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:14.468635 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7e4a4c7-ef99-479f-91b0-97a25df06c9a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:14.470065 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7e4a4c7-ef99-479f-91b0-97a25df06c9a None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:14.483505 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 221/437] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Oct 28 09:31:14 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:31:14.484652 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a331a16e-db41-4200-a6ed-cfef9fa43fe5 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:14.484950 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a331a16e-db41-4200-a6ed-cfef9fa43fe5 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:14.485228 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a331a16e-db41-4200-a6ed-cfef9fa43fe5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:14.503238 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-a331a16e-db41-4200-a6ed-cfef9fa43fe5 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:14.504415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a331a16e-db41-4200-a6ed-cfef9fa43fe5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:14.559149 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b145ac61-b2ae-42fb-ae20-76d546bbbdb2 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:14.590437 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 222/438] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:31:14.590437 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a331a16e-db41-4200-a6ed-cfef9fa43fe5 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:14.598464 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:14.603885 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-924f6fc7-200a-4197-b1e6-14f2a5bcb235 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:14.604870 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924f6fc7-200a-4197-b1e6-14f2a5bcb235 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:14.605227 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924f6fc7-200a-4197-b1e6-14f2a5bcb235 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:14.605683 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924f6fc7-200a-4197-b1e6-14f2a5bcb235 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:14.606454 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-924f6fc7-200a-4197-b1e6-14f2a5bcb235 None admin] RBAC: Adding query filter params (group.id=71818c8c50704a6ca8dfade2fa6e5a7d, role.id=d93b4c94b590457199c3d301f947b0fb, scope.project.id=6e28083c55ff4290af640f011fd2466f) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:14.606773 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-924f6fc7-200a-4197-b1e6-14f2a5bcb235 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:14.608410 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-924f6fc7-200a-4197-b1e6-14f2a5bcb235 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:14.623014 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 217/439] 10.222.0.25 () {56 vars in 1376 bytes} [Fri Oct 28 09:31:14 2022] GET /identity/v3/role_assignments?group.id=71818c8c50704a6ca8dfade2fa6e5a7d&scope.project.id=6e28083c55ff4290af640f011fd2466f&role.id=d93b4c94b590457199c3d301f947b0fb => generated 611 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:14.673906 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:14.708216 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a331a16e-db41-4200-a6ed-cfef9fa43fe5 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:14.719150 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-924f6fc7-200a-4197-b1e6-14f2a5bcb235 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:14.760116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f5477b-6c59-42e2-9e71-8dc4baafd071 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:14.760739 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f5477b-6c59-42e2-9e71-8dc4baafd071 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:14.761015 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f5477b-6c59-42e2-9e71-8dc4baafd071 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:14.761285 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f5477b-6c59-42e2-9e71-8dc4baafd071 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:14.836548 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2f5477b-6c59-42e2-9e71-8dc4baafd071 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:14.837988 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2f5477b-6c59-42e2-9e71-8dc4baafd071 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:14.854498 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-924f6fc7-200a-4197-b1e6-14f2a5bcb235 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:14.896460 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f4e160-c965-419e-8f97-763bb233aea9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:14.897226 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f4e160-c965-419e-8f97-763bb233aea9 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:14.897782 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f4e160-c965-419e-8f97-763bb233aea9 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:14.898125 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f4e160-c965-419e-8f97-763bb233aea9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:14.909886 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f4e160-c965-419e-8f97-763bb233aea9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:31:14.915741 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-f0f4e160-c965-419e-8f97-763bb233aea9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:31:14.915741 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:14.915741 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:14.915741 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:14.915741 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:14.915741 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:14.915741 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:14.915741 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:14.915741 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:14.915741 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:14.915741 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:14.915741 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:14.915741 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:31:14.915741 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:31:14.915741 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:31:14.915741 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:31:14.916692 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:31:14.916692 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:31:14.916692 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:31:14.916692 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:31:14.916692 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:14.916692 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:14.916692 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:14.916692 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:14.916692 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:14.916692 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:14.916692 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:14.916692 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:14.916692 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:14.916692 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:14.916692 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:14.916692 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:14.917483 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:14.917483 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:14.917483 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:14.917483 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:14.917483 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:31:14.917483 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:31:14.917483 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:31:14.917483 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:31:14.917483 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:31:14.917483 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:31:14.917483 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:31:14.917483 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:14.917483 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 218/440] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Oct 28 09:31:14 2022] GET /identity/v3/roles/admin => generated 83 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:31:14.918164 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f4e160-c965-419e-8f97-763bb233aea9 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:14.921622 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:14.953514 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 223/441] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:31:14 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 09:31:14.954365 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f5477b-6c59-42e2-9e71-8dc4baafd071 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:14.979610 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:15.022839 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f4e160-c965-419e-8f97-763bb233aea9 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:15.052807 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f5477b-6c59-42e2-9e71-8dc4baafd071 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:15.056411 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-340c20b6-a364-41d0-badf-f78de8025529 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:15.057045 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340c20b6-a364-41d0-badf-f78de8025529 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:15.057348 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340c20b6-a364-41d0-badf-f78de8025529 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:15.057620 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340c20b6-a364-41d0-badf-f78de8025529 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:15.067544 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-340c20b6-a364-41d0-badf-f78de8025529 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:31:15.072894 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-340c20b6-a364-41d0-badf-f78de8025529 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 09:31:15.072894 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:15.072894 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:15.072894 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:15.072894 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:15.072894 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:15.072894 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:15.072894 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:15.072894 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:15.072894 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:15.072894 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:15.072894 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:15.072894 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:31:15.072894 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:31:15.072894 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:31:15.072894 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:31:15.073826 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:31:15.073826 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:31:15.073826 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:31:15.073826 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:31:15.073826 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:15.073826 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:15.073826 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:15.073826 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:15.073826 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:15.073826 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:15.073826 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:15.073826 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:15.073826 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:15.073826 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:15.073826 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:15.073826 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:15.074659 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:15.074659 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:15.074659 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:15.074659 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:15.074659 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:31:15.074659 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:31:15.074659 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:31:15.074659 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:31:15.074659 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:31:15.074659 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:31:15.074659 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 09:31:15.074659 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:15.074659 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 219/442] 10.222.0.25 () {56 vars in 1114 bytes} [Fri Oct 28 09:31:14 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:15.075597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-340c20b6-a364-41d0-badf-f78de8025529 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:15.090648 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64bc5798-fd17-4d71-a90f-7e7d44a34846 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:15.091259 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64bc5798-fd17-4d71-a90f-7e7d44a34846 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:15.091540 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64bc5798-fd17-4d71-a90f-7e7d44a34846 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:15.091816 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64bc5798-fd17-4d71-a90f-7e7d44a34846 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:15.092476 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64bc5798-fd17-4d71-a90f-7e7d44a34846 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:15.092778 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64bc5798-fd17-4d71-a90f-7e7d44a34846 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:15.094144 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64bc5798-fd17-4d71-a90f-7e7d44a34846 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:15.102841 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 224/443] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:31:14 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:15.102841 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64bc5798-fd17-4d71-a90f-7e7d44a34846 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:15.136123 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:15.178341 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-340c20b6-a364-41d0-badf-f78de8025529 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:15.194317 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:15.204916 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64bc5798-fd17-4d71-a90f-7e7d44a34846 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:15.212257 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d26168f2-d403-410b-a067-28536f7578f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:15.212889 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26168f2-d403-410b-a067-28536f7578f4 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:15.213163 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26168f2-d403-410b-a067-28536f7578f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:15.213432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26168f2-d403-410b-a067-28536f7578f4 None admin] PATH_INFO: `/v3/projects/6e28083c55ff4290af640f011fd2466f` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:15.214601 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d26168f2-d403-410b-a067-28536f7578f4 None admin] RBAC: Authorizing `identity:get_project(project_id=6e28083c55ff4290af640f011fd2466f)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:15.216034 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d26168f2-d403-410b-a067-28536f7578f4 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:15.217627 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 220/444] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:31:15 2022] GET /identity/v3/projects/6e28083c55ff4290af640f011fd2466f => generated 298 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:15.218380 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d26168f2-d403-410b-a067-28536f7578f4 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:15.243382 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce1efcb-c955-4377-94f6-0034f71d55f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:15.243909 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce1efcb-c955-4377-94f6-0034f71d55f8 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:15.244360 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce1efcb-c955-4377-94f6-0034f71d55f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:15.244701 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce1efcb-c955-4377-94f6-0034f71d55f8 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:15.245416 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce1efcb-c955-4377-94f6-0034f71d55f8 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:15.245719 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce1efcb-c955-4377-94f6-0034f71d55f8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:15.247115 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce1efcb-c955-4377-94f6-0034f71d55f8 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:15.255661 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 225/445] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:31:15 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:15.256454 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce1efcb-c955-4377-94f6-0034f71d55f8 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:15.316259 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d26168f2-d403-410b-a067-28536f7578f4 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:15.349059 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-663ad7a7-91d3-4904-8388-c191f4f3d928 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:15.349665 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663ad7a7-91d3-4904-8388-c191f4f3d928 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:15.349941 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663ad7a7-91d3-4904-8388-c191f4f3d928 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:15.350211 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663ad7a7-91d3-4904-8388-c191f4f3d928 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:15.355052 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:15.361899 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce1efcb-c955-4377-94f6-0034f71d55f8 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:15.397544 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dbc94d3-4bdb-45b9-92ff-66af21d6d364 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:15.398513 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dbc94d3-4bdb-45b9-92ff-66af21d6d364 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:15.398798 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dbc94d3-4bdb-45b9-92ff-66af21d6d364 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:15.399071 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dbc94d3-4bdb-45b9-92ff-66af21d6d364 None admin] PATH_INFO: `/v3/groups/71818c8c50704a6ca8dfade2fa6e5a7d` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:15.400186 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dbc94d3-4bdb-45b9-92ff-66af21d6d364 None admin] RBAC: Authorizing `identity:get_group(group_id=71818c8c50704a6ca8dfade2fa6e5a7d)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:15.400941 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:15.402102 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dbc94d3-4bdb-45b9-92ff-66af21d6d364 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:15.404145 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 226/446] 10.222.0.25 () {56 vars in 1180 bytes} [Fri Oct 28 09:31:15 2022] GET /identity/v3/groups/71818c8c50704a6ca8dfade2fa6e5a7d => generated 226 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:15.404912 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dbc94d3-4bdb-45b9-92ff-66af21d6d364 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:15.417595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-663ad7a7-91d3-4904-8388-c191f4f3d928 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:15.419094 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-663ad7a7-91d3-4904-8388-c191f4f3d928 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:15.507865 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dbc94d3-4bdb-45b9-92ff-66af21d6d364 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:15.519248 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 221/447] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:31:15 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 09:31:15.520108 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-663ad7a7-91d3-4904-8388-c191f4f3d928 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:15.543524 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd048f9-304e-43f2-9899-5b3553996928 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:15.544140 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd048f9-304e-43f2-9899-5b3553996928 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:15.544421 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd048f9-304e-43f2-9899-5b3553996928 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:15.544697 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd048f9-304e-43f2-9899-5b3553996928 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:15.578223 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:15.607219 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:15.617647 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bd048f9-304e-43f2-9899-5b3553996928 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:15.619144 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bd048f9-304e-43f2-9899-5b3553996928 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:15.623461 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-663ad7a7-91d3-4904-8388-c191f4f3d928 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:15.656666 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2657c4dc-20cc-4ab8-8caf-a54f82e26721 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:15.657275 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2657c4dc-20cc-4ab8-8caf-a54f82e26721 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:15.657813 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2657c4dc-20cc-4ab8-8caf-a54f82e26721 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:15.658092 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2657c4dc-20cc-4ab8-8caf-a54f82e26721 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:15.658851 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2657c4dc-20cc-4ab8-8caf-a54f82e26721 None admin] RBAC: Adding query filter params (group.id=71818c8c50704a6ca8dfade2fa6e5a7d, role.id=8943f289c9fe4feba2695b9c90cf3d51, scope.project.id=6e28083c55ff4290af640f011fd2466f) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:15.659166 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2657c4dc-20cc-4ab8-8caf-a54f82e26721 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:15.660580 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2657c4dc-20cc-4ab8-8caf-a54f82e26721 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:15.666781 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 222/448] 10.222.0.25 () {56 vars in 1376 bytes} [Fri Oct 28 09:31:15 2022] GET /identity/v3/role_assignments?group.id=71818c8c50704a6ca8dfade2fa6e5a7d&scope.project.id=6e28083c55ff4290af640f011fd2466f&role.id=8943f289c9fe4feba2695b9c90cf3d51 => generated 611 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:15.667564 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2657c4dc-20cc-4ab8-8caf-a54f82e26721 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:15.739426 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 227/449] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:31:15 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 335 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 09:31:15.745796 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd048f9-304e-43f2-9899-5b3553996928 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:15.781920 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2657c4dc-20cc-4ab8-8caf-a54f82e26721 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:15.816171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:15.818717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ecd9d2-ba30-49db-8793-d2c7acfea0c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:15.819377 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ecd9d2-ba30-49db-8793-d2c7acfea0c5 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:15.819661 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ecd9d2-ba30-49db-8793-d2c7acfea0c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:15.819964 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ecd9d2-ba30-49db-8793-d2c7acfea0c5 None admin] PATH_INFO: `/v3/projects/ac54d153685c4b2288a7ed215d0e24d2` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:15.821164 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23ecd9d2-ba30-49db-8793-d2c7acfea0c5 None admin] RBAC: Authorizing `identity:get_project(project_id=ac54d153685c4b2288a7ed215d0e24d2)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:15.822683 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23ecd9d2-ba30-49db-8793-d2c7acfea0c5 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:15.824571 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 223/450] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:31:15 2022] GET /identity/v3/projects/ac54d153685c4b2288a7ed215d0e24d2 => generated 344 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:15.830038 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ecd9d2-ba30-49db-8793-d2c7acfea0c5 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:15.851290 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:15.863240 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd048f9-304e-43f2-9899-5b3553996928 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:15.905663 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f71a38eb-2e63-452a-a281-dec4c57aeb88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:15.906359 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f71a38eb-2e63-452a-a281-dec4c57aeb88 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:15.906679 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f71a38eb-2e63-452a-a281-dec4c57aeb88 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:15.906982 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f71a38eb-2e63-452a-a281-dec4c57aeb88 None admin] PATH_INFO: `/v3/projects/5960af42464f4e83b60ec52264d04b70` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:15.908415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f71a38eb-2e63-452a-a281-dec4c57aeb88 None admin] RBAC: Authorizing `identity:get_project(project_id=5960af42464f4e83b60ec52264d04b70)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:15.909843 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f71a38eb-2e63-452a-a281-dec4c57aeb88 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:15.911530 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 228/451] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:31:15 2022] GET /identity/v3/projects/5960af42464f4e83b60ec52264d04b70 => generated 302 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:31:15.917129 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f71a38eb-2e63-452a-a281-dec4c57aeb88 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:15.954279 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ecd9d2-ba30-49db-8793-d2c7acfea0c5 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:15.992845 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71efba93-b359-4014-818e-44a5c65526ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:15.993633 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71efba93-b359-4014-818e-44a5c65526ae None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:15.993909 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71efba93-b359-4014-818e-44a5c65526ae None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:15.994182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71efba93-b359-4014-818e-44a5c65526ae None admin] PATH_INFO: `/v3/groups/1c414fe7bf464383a9eee4581f8f3c2f` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:15.995360 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71efba93-b359-4014-818e-44a5c65526ae None admin] RBAC: Authorizing `identity:get_group(group_id=1c414fe7bf464383a9eee4581f8f3c2f)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:15.996812 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71efba93-b359-4014-818e-44a5c65526ae None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:15.998450 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 224/452] 10.222.0.25 () {56 vars in 1180 bytes} [Fri Oct 28 09:31:15 2022] GET /identity/v3/groups/1c414fe7bf464383a9eee4581f8f3c2f => generated 229 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:31:16.004389 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71efba93-b359-4014-818e-44a5c65526ae None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:16.035612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f71a38eb-2e63-452a-a281-dec4c57aeb88 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:16.071368 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:16.079290 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8312e82c-7baf-4205-9462-0b49f6c4762e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:16.080109 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8312e82c-7baf-4205-9462-0b49f6c4762e None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:16.080391 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8312e82c-7baf-4205-9462-0b49f6c4762e None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:16.081551 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8312e82c-7baf-4205-9462-0b49f6c4762e None admin] PATH_INFO: `/v3/groups/71818c8c50704a6ca8dfade2fa6e5a7d` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:16.082415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8312e82c-7baf-4205-9462-0b49f6c4762e None admin] RBAC: Authorizing `identity:get_group(group_id=71818c8c50704a6ca8dfade2fa6e5a7d)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:16.083847 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8312e82c-7baf-4205-9462-0b49f6c4762e None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:16.085952 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 229/453] 10.222.0.25 () {56 vars in 1180 bytes} [Fri Oct 28 09:31:15 2022] GET /identity/v3/groups/71818c8c50704a6ca8dfade2fa6e5a7d => generated 226 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:31:16.092041 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8312e82c-7baf-4205-9462-0b49f6c4762e None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:16.104011 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:16.124457 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71efba93-b359-4014-818e-44a5c65526ae None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:16.169705 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ffea4c-7a7c-4dfa-91b8-00eecaa86fb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:16.170373 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ffea4c-7a7c-4dfa-91b8-00eecaa86fb0 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:16.170753 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ffea4c-7a7c-4dfa-91b8-00eecaa86fb0 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:16.171534 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ffea4c-7a7c-4dfa-91b8-00eecaa86fb0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:16.172293 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ffea4c-7a7c-4dfa-91b8-00eecaa86fb0 None admin] RBAC: Adding query filter params (group.id=1c414fe7bf464383a9eee4581f8f3c2f, role.id=e13d798a98ea4c3dbb87a236ec683d32, scope.project.id=ac54d153685c4b2288a7ed215d0e24d2) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:16.172759 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ffea4c-7a7c-4dfa-91b8-00eecaa86fb0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:16.174234 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ffea4c-7a7c-4dfa-91b8-00eecaa86fb0 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:16.180351 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 225/454] 10.222.0.25 () {56 vars in 1376 bytes} [Fri Oct 28 09:31:16 2022] GET /identity/v3/role_assignments?group.id=1c414fe7bf464383a9eee4581f8f3c2f&scope.project.id=ac54d153685c4b2288a7ed215d0e24d2&role.id=e13d798a98ea4c3dbb87a236ec683d32 => generated 611 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:31:16.211422 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8312e82c-7baf-4205-9462-0b49f6c4762e None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:16.255084 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab9d9e3-60f8-4fdc-abb0-6ff0ce2b9b30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:16.255709 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab9d9e3-60f8-4fdc-abb0-6ff0ce2b9b30 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:16.255987 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab9d9e3-60f8-4fdc-abb0-6ff0ce2b9b30 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:16.256258 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab9d9e3-60f8-4fdc-abb0-6ff0ce2b9b30 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:16.257009 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ab9d9e3-60f8-4fdc-abb0-6ff0ce2b9b30 None admin] RBAC: Adding query filter params (group.id=71818c8c50704a6ca8dfade2fa6e5a7d, role.id=d93b4c94b590457199c3d301f947b0fb, scope.project.id=5960af42464f4e83b60ec52264d04b70) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:16.257356 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ab9d9e3-60f8-4fdc-abb0-6ff0ce2b9b30 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:16.258938 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ab9d9e3-60f8-4fdc-abb0-6ff0ce2b9b30 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:16.266150 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 230/455] 10.222.0.25 () {56 vars in 1376 bytes} [Fri Oct 28 09:31:16 2022] GET /identity/v3/role_assignments?group.id=71818c8c50704a6ca8dfade2fa6e5a7d&scope.project.id=5960af42464f4e83b60ec52264d04b70&role.id=d93b4c94b590457199c3d301f947b0fb => generated 611 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:31:18.983316 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec48a41-8b94-46c3-81c8-c443e0101b5c None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:18.984353 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec48a41-8b94-46c3-81c8-c443e0101b5c None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:18.984837 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec48a41-8b94-46c3-81c8-c443e0101b5c None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:18.986655 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 226/456] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:18.992973 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b8f531-0b69-478a-b122-8a9337909b57 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:18.993848 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b8f531-0b69-478a-b122-8a9337909b57 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:18.994317 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b8f531-0b69-478a-b122-8a9337909b57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:19.024215 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-50b8f531-0b69-478a-b122-8a9337909b57 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:19.025842 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-50b8f531-0b69-478a-b122-8a9337909b57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:19.134785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:19.138316 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 231/457] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:31:19.149509 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d0be26-8a75-475b-8d4d-dfd1f8e1ea09 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:19.149788 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d0be26-8a75-475b-8d4d-dfd1f8e1ea09 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:19.150073 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d0be26-8a75-475b-8d4d-dfd1f8e1ea09 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:19.151673 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 227/458] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:19 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:19.157172 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fefa3d09-e844-4206-b9e5-be219a9a021b None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:19.157716 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fefa3d09-e844-4206-b9e5-be219a9a021b None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:19.157991 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fefa3d09-e844-4206-b9e5-be219a9a021b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:19.187531 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-fefa3d09-e844-4206-b9e5-be219a9a021b None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:19.188732 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-fefa3d09-e844-4206-b9e5-be219a9a021b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:19.280095 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 232/459] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:31:19.517456 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d0be26-8a75-475b-8d4d-dfd1f8e1ea09 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:19.645092 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24d0be26-8a75-475b-8d4d-dfd1f8e1ea09 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:19.683448 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-568f2166-1480-4c08-b2cc-b6cda4e375a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:19.684089 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568f2166-1480-4c08-b2cc-b6cda4e375a8 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:19.684370 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568f2166-1480-4c08-b2cc-b6cda4e375a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:19.684647 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568f2166-1480-4c08-b2cc-b6cda4e375a8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:19.695818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-568f2166-1480-4c08-b2cc-b6cda4e375a8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:19.697286 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-568f2166-1480-4c08-b2cc-b6cda4e375a8 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:19.699982 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e249ab-d228-4b59-8f0d-67547cf1e18c None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:19.700278 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e249ab-d228-4b59-8f0d-67547cf1e18c None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:19.700562 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e249ab-d228-4b59-8f0d-67547cf1e18c None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:19.702164 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 233/460] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:19.707004 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:19.707497 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6884ae2-aa18-4bd3-bd33-d948f3482929 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:19.707785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6884ae2-aa18-4bd3-bd33-d948f3482929 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:19.708071 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6884ae2-aa18-4bd3-bd33-d948f3482929 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:19.714617 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-568f2166-1480-4c08-b2cc-b6cda4e375a8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:19.714617 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:19.714617 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:19.714617 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:19.714617 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:19.714617 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:19.714617 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:19.714617 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:19.714617 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:19.714617 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:19.714617 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:19.714617 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:19.714617 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:19.714617 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:19.714617 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:19.714617 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:19.715530 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:19.715530 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:19.715530 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:19.715530 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:19.715530 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:19.715530 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:19.715530 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:19.715530 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:19.715530 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:19.715530 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:19.715530 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:19.715530 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:19.715530 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:19.715530 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:19.715530 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:19.715530 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:19.715530 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:19.716367 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:19.716367 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:19.716367 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:19.718370 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 228/461] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:19.726523 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-568f2166-1480-4c08-b2cc-b6cda4e375a8 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:19.746602 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-b6884ae2-aa18-4bd3-bd33-d948f3482929 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:19.747806 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-b6884ae2-aa18-4bd3-bd33-d948f3482929 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:19.841544 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 234/462] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:31:19.842851 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6884ae2-aa18-4bd3-bd33-d948f3482929 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:19.874759 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-568f2166-1480-4c08-b2cc-b6cda4e375a8 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:19.910860 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550085ba-bf52-4e70-a3c4-284a3c006dfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:19.911792 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550085ba-bf52-4e70-a3c4-284a3c006dfc None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:19.912248 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550085ba-bf52-4e70-a3c4-284a3c006dfc None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:19.912821 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:19.914129 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550085ba-bf52-4e70-a3c4-284a3c006dfc None admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:19.915071 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550085ba-bf52-4e70-a3c4-284a3c006dfc None admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:19.915615 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550085ba-bf52-4e70-a3c4-284a3c006dfc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:19.917196 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550085ba-bf52-4e70-a3c4-284a3c006dfc None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:19.937158 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 229/463] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:19.942829 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550085ba-bf52-4e70-a3c4-284a3c006dfc None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:20.005430 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6884ae2-aa18-4bd3-bd33-d948f3482929 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:20.037246 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:20.048518 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8c30dac-2b08-4050-974a-ce60ca9a5fe6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:20.049371 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c30dac-2b08-4050-974a-ce60ca9a5fe6 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:20.049902 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c30dac-2b08-4050-974a-ce60ca9a5fe6 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:20.050322 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c30dac-2b08-4050-974a-ce60ca9a5fe6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:20.064726 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8c30dac-2b08-4050-974a-ce60ca9a5fe6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:20.066615 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8c30dac-2b08-4050-974a-ce60ca9a5fe6 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:20.089487 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-d8c30dac-2b08-4050-974a-ce60ca9a5fe6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:20.089487 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:20.089487 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:20.089487 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:20.089487 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:20.089487 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:20.089487 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:20.089487 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:20.089487 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:20.089487 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:20.089487 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:20.089487 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:20.089487 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:20.089487 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:20.089487 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:20.089487 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:20.090460 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:20.090460 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:20.090460 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:20.090460 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:20.090460 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:20.090460 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:20.090460 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:20.090460 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:20.090460 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:20.090460 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:20.090460 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:20.090460 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:20.090460 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:20.090460 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:20.090460 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:20.090460 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:20.090460 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:20.091335 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:20.091335 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:20.091335 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:20.091335 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 235/464] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:20.092170 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550085ba-bf52-4e70-a3c4-284a3c006dfc None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:20.092323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c7398d-050c-417c-9a30-a0a28eff1eaa None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:20.092780 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c7398d-050c-417c-9a30-a0a28eff1eaa None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:20.093192 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c7398d-050c-417c-9a30-a0a28eff1eaa None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:20.095436 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 236/465] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:20.096295 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c7398d-050c-417c-9a30-a0a28eff1eaa None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:20.133021 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d308bc-43d3-447f-a43e-65f6d71715d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:20.133804 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d308bc-43d3-447f-a43e-65f6d71715d5 None admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:20.134183 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d308bc-43d3-447f-a43e-65f6d71715d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:20.134596 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d308bc-43d3-447f-a43e-65f6d71715d5 None admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:20.135365 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92d308bc-43d3-447f-a43e-65f6d71715d5 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:20.136910 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92d308bc-43d3-447f-a43e-65f6d71715d5 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:20.161287 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 230/466] 10.222.0.25 () {60 vars in 1132 bytes} [Fri Oct 28 09:31:19 2022] POST /identity/v3/users => generated 246 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:31:20.163376 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765b0690-9b7f-4f33-bffe-d60b9e88af18 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:20.163376 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765b0690-9b7f-4f33-bffe-d60b9e88af18 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:20.163602 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765b0690-9b7f-4f33-bffe-d60b9e88af18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:20.185680 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-765b0690-9b7f-4f33-bffe-d60b9e88af18 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:20.187037 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-765b0690-9b7f-4f33-bffe-d60b9e88af18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:20.222506 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c7398d-050c-417c-9a30-a0a28eff1eaa None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:20.227159 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:20.267071 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c5af064-26e3-4213-8cfd-135821ab4deb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:20.268336 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5af064-26e3-4213-8cfd-135821ab4deb None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:20.269276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5af064-26e3-4213-8cfd-135821ab4deb None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:20.270041 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5af064-26e3-4213-8cfd-135821ab4deb None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:20.270984 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c5af064-26e3-4213-8cfd-135821ab4deb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:20.271498 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c5af064-26e3-4213-8cfd-135821ab4deb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:20.273624 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c5af064-26e3-4213-8cfd-135821ab4deb None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:20.278864 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 231/467] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:31:20.279872 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-765b0690-9b7f-4f33-bffe-d60b9e88af18 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:20.288170 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 237/468] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:20.298615 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c5af064-26e3-4213-8cfd-135821ab4deb None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:20.403809 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:20.475514 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-765b0690-9b7f-4f33-bffe-d60b9e88af18 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:20.510040 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c5af064-26e3-4213-8cfd-135821ab4deb None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:20.550542 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0697ed5-0227-4642-b38e-90ba18a02a74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:20.551198 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0697ed5-0227-4642-b38e-90ba18a02a74 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:20.551479 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0697ed5-0227-4642-b38e-90ba18a02a74 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:20.551750 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0697ed5-0227-4642-b38e-90ba18a02a74 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:20.565988 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f6fad5-3f05-4834-921a-635b77779c99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:20.566638 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f6fad5-3f05-4834-921a-635b77779c99 None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:20.566924 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f6fad5-3f05-4834-921a-635b77779c99 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:20.567267 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f6fad5-3f05-4834-921a-635b77779c99 None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:20.567935 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f6fad5-3f05-4834-921a-635b77779c99 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:20.569335 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0697ed5-0227-4642-b38e-90ba18a02a74 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:20.569520 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f6fad5-3f05-4834-921a-635b77779c99 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:20.571381 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0697ed5-0227-4642-b38e-90ba18a02a74 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:20.600190 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 238/469] 10.222.0.25 () {60 vars in 1132 bytes} [Fri Oct 28 09:31:20 2022] POST /identity/v3/users => generated 248 bytes in 302 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:31:20.604590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-a0697ed5-0227-4642-b38e-90ba18a02a74 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:20.604590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:20.604590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:20.604590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:20.604590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:20.604590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:20.604590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:20.604590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:20.604590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:20.604590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:20.604590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:20.604590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:20.604590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:20.604590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:20.604590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:20.604590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:20.605541 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:20.605541 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:20.605541 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:20.605541 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:20.605541 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:20.605541 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:20.605541 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:20.605541 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:20.605541 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:20.605541 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:20.605541 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:20.605541 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:20.605541 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:20.605541 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:20.605541 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:20.605541 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:20.605541 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:20.606436 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:20.606436 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:20.606436 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:20.606436 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 232/470] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 327 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:20.613321 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f6fad5-3f05-4834-921a-635b77779c99 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:20.631309 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0697ed5-0227-4642-b38e-90ba18a02a74 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:20.648250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:20.777001 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f6fad5-3f05-4834-921a-635b77779c99 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:20.806942 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:20.814179 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0697ed5-0227-4642-b38e-90ba18a02a74 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:20.821251 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f36ecf82-fc5b-4d80-b1d3-c1683c69bd78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:20.821885 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36ecf82-fc5b-4d80-b1d3-c1683c69bd78 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:20.822174 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36ecf82-fc5b-4d80-b1d3-c1683c69bd78 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:20.822478 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f36ecf82-fc5b-4d80-b1d3-c1683c69bd78 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:20.823143 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f36ecf82-fc5b-4d80-b1d3-c1683c69bd78 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:20.823443 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f36ecf82-fc5b-4d80-b1d3-c1683c69bd78 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:20.824841 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f36ecf82-fc5b-4d80-b1d3-c1683c69bd78 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:20.839241 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 239/471] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:20.840547 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2eb262-a839-417d-b1ac-2a0e12051bf6 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:20.840825 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2eb262-a839-417d-b1ac-2a0e12051bf6 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:20.841096 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2eb262-a839-417d-b1ac-2a0e12051bf6 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:20.851392 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 240/472] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:20 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:20.851392 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2eb262-a839-417d-b1ac-2a0e12051bf6 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:20.873073 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-892f39b1-19fe-492b-a79a-0372b021faf5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:20.874372 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-892f39b1-19fe-492b-a79a-0372b021faf5 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:20.874897 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-892f39b1-19fe-492b-a79a-0372b021faf5 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:20.875454 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-892f39b1-19fe-492b-a79a-0372b021faf5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:20.887617 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-892f39b1-19fe-492b-a79a-0372b021faf5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:20.890423 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-892f39b1-19fe-492b-a79a-0372b021faf5 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:20.909448 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-892f39b1-19fe-492b-a79a-0372b021faf5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:20.909448 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:20.909448 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:20.909448 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:20.909448 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:20.909448 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:20.909448 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:20.909448 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:20.909448 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:20.909448 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:20.909448 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:20.909448 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:20.909448 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:20.909448 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:20.909448 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:20.909448 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:20.910453 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:20.910453 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:20.910453 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:20.910453 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:20.910453 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:20.910453 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:20.910453 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:20.910453 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:20.910453 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:20.910453 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:20.910453 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:20.910453 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:20.910453 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:20.910453 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:20.910453 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:20.910453 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:20.910453 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:20.911320 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:20.911320 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:20.911320 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:20.914183 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 233/473] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 283 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:20.916231 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077b1882-14b6-450b-abdc-328d3c780d8e None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:20.916734 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077b1882-14b6-450b-abdc-328d3c780d8e None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:20.917253 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077b1882-14b6-450b-abdc-328d3c780d8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:20.939526 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-077b1882-14b6-450b-abdc-328d3c780d8e None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:20.941073 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-077b1882-14b6-450b-abdc-328d3c780d8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:20.946028 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:20.968614 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2eb262-a839-417d-b1ac-2a0e12051bf6 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:21.014012 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982d7131-6274-4f5a-9bc4-41dde8ec7578 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:21.014012 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982d7131-6274-4f5a-9bc4-41dde8ec7578 None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:21.014012 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982d7131-6274-4f5a-9bc4-41dde8ec7578 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:21.014827 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982d7131-6274-4f5a-9bc4-41dde8ec7578 None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:21.015069 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-982d7131-6274-4f5a-9bc4-41dde8ec7578 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:21.016745 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-982d7131-6274-4f5a-9bc4-41dde8ec7578 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:21.042222 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 234/474] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:31:21.045820 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 241/475] 10.222.0.25 () {60 vars in 1132 bytes} [Fri Oct 28 09:31:20 2022] POST /identity/v3/users => generated 248 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:31:21.047266 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077b1882-14b6-450b-abdc-328d3c780d8e None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:21.141367 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:21.163941 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077b1882-14b6-450b-abdc-328d3c780d8e None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:21.212418 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96e15cd-6f20-4904-aac9-49ffbf8120e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:21.213042 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96e15cd-6f20-4904-aac9-49ffbf8120e9 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:21.213324 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96e15cd-6f20-4904-aac9-49ffbf8120e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:21.213597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96e15cd-6f20-4904-aac9-49ffbf8120e9 None admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:21.214245 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b96e15cd-6f20-4904-aac9-49ffbf8120e9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:21.214576 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b96e15cd-6f20-4904-aac9-49ffbf8120e9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:21.215955 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b96e15cd-6f20-4904-aac9-49ffbf8120e9 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:21.232426 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 235/476] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:21.238454 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982d7131-6274-4f5a-9bc4-41dde8ec7578 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:21.365710 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982d7131-6274-4f5a-9bc4-41dde8ec7578 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:21.400997 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f01724b-e80e-4f8d-b299-4e2392dfe12e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:21.401608 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f01724b-e80e-4f8d-b299-4e2392dfe12e None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:21.401888 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f01724b-e80e-4f8d-b299-4e2392dfe12e None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:21.402159 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f01724b-e80e-4f8d-b299-4e2392dfe12e None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:21.402852 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f01724b-e80e-4f8d-b299-4e2392dfe12e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:21.404385 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f01724b-e80e-4f8d-b299-4e2392dfe12e None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:21.410337 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96e15cd-6f20-4904-aac9-49ffbf8120e9 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:21.415064 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:21.435050 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 242/477] 10.222.0.25 () {60 vars in 1132 bytes} [Fri Oct 28 09:31:21 2022] POST /identity/v3/users => generated 249 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:31:21.542562 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96e15cd-6f20-4904-aac9-49ffbf8120e9 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:21.584257 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a473fb-cd28-43fd-91d3-f171bf86755e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:21.584890 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a473fb-cd28-43fd-91d3-f171bf86755e None admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:21.585192 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a473fb-cd28-43fd-91d3-f171bf86755e None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:21.586062 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a473fb-cd28-43fd-91d3-f171bf86755e None admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:21.586853 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a473fb-cd28-43fd-91d3-f171bf86755e None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:21.588407 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a473fb-cd28-43fd-91d3-f171bf86755e None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:21.598541 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.sql.core [None req-d6a473fb-cd28-43fd-91d3-f171bf86755e None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 28 09:31:21.598541 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Oct 28 09:31:21.598541 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [parameters: {'id': 'bc3fa182e14f44f19859ce63c7354d1c', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Oct 28 09:31:21.598541 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81576) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 28 09:31:21.617463 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-d6a473fb-cd28-43fd-91d3-f171bf86755e None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Oct 28 09:31:21.617463 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:21.617463 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 28 09:31:21.617463 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 28 09:31:21.617463 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 28 09:31:21.617463 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 28 09:31:21.617463 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 28 09:31:21.617463 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application result = self._query(query) Oct 28 09:31:21.617463 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 28 09:31:21.617463 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application conn.query(q) Oct 28 09:31:21.617463 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 28 09:31:21.617463 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 28 09:31:21.617463 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 28 09:31:21.617463 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application result.read() Oct 28 09:31:21.617463 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 28 09:31:21.618348 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 28 09:31:21.618348 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 28 09:31:21.618348 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 28 09:31:21.618348 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 28 09:31:21.618348 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 28 09:31:21.618348 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 28 09:31:21.618348 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 28 09:31:21.618348 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 28 09:31:21.618348 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:21.618348 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 28 09:31:21.618348 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:21.618348 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:21.618348 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 28 09:31:21.618348 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 28 09:31:21.618348 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Oct 28 09:31:21.618348 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return ref.to_dict() Oct 28 09:31:21.618348 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 28 09:31:21.618348 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application next(self.gen) Oct 28 09:31:21.620152 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 28 09:31:21.620152 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application yield resource Oct 28 09:31:21.620152 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 28 09:31:21.620152 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application next(self.gen) Oct 28 09:31:21.620152 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 28 09:31:21.620152 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application self.session.rollback() Oct 28 09:31:21.620152 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 28 09:31:21.620152 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application self.force_reraise() Oct 28 09:31:21.620152 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 28 09:31:21.620152 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise self.value Oct 28 09:31:21.620152 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 28 09:31:21.620152 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 28 09:31:21.620152 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 28 09:31:21.620152 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application session.commit() Oct 28 09:31:21.620152 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 28 09:31:21.620152 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 28 09:31:21.620152 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 28 09:31:21.622205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application self._prepare_impl() Oct 28 09:31:21.622205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 28 09:31:21.622205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application self.session.flush() Oct 28 09:31:21.622205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 28 09:31:21.622205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application self._flush(objects) Oct 28 09:31:21.622205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 28 09:31:21.622205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 28 09:31:21.622205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 28 09:31:21.622205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application compat.raise_( Oct 28 09:31:21.622205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 28 09:31:21.622205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception Oct 28 09:31:21.622205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 28 09:31:21.622205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application flush_context.execute() Oct 28 09:31:21.622205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 28 09:31:21.622205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rec.execute(self) Oct 28 09:31:21.622205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 28 09:31:21.622205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 28 09:31:21.628304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 28 09:31:21.628304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 28 09:31:21.628304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 28 09:31:21.628304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 28 09:31:21.628304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 28 09:31:21.628304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 28 09:31:21.628304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 28 09:31:21.628304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 28 09:31:21.628304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 28 09:31:21.628304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 28 09:31:21.628304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 28 09:31:21.628304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 28 09:31:21.628304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 28 09:31:21.628304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 28 09:31:21.628304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 28 09:31:21.628304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception Oct 28 09:31:21.630165 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 28 09:31:21.630165 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 28 09:31:21.630165 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 28 09:31:21.630165 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 28 09:31:21.630165 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 28 09:31:21.630165 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application result = self._query(query) Oct 28 09:31:21.630165 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 28 09:31:21.630165 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application conn.query(q) Oct 28 09:31:21.630165 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 28 09:31:21.630165 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 28 09:31:21.630165 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 28 09:31:21.630165 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application result.read() Oct 28 09:31:21.630165 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 28 09:31:21.630165 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 28 09:31:21.630165 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 28 09:31:21.630165 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 28 09:31:21.632324 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 28 09:31:21.632324 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 28 09:31:21.632324 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 28 09:31:21.632324 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 28 09:31:21.632324 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Oct 28 09:31:21.632324 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Oct 28 09:31:21.632324 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [parameters: {'id': 'bc3fa182e14f44f19859ce63c7354d1c', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Oct 28 09:31:21.632324 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 28 09:31:21.632324 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:21.632324 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 09:31:21.632324 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:21.632324 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:21.632324 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:21.632324 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:21.632324 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:21.632324 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:21.634309 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:21.634309 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:21.634309 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:21.634309 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:21.634309 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:21.634309 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:21.634309 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Oct 28 09:31:21.634309 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Oct 28 09:31:21.634309 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:21.634309 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:21.634309 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Oct 28 09:31:21.634309 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Oct 28 09:31:21.634309 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 28 09:31:21.634309 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 28 09:31:21.634309 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Oct 28 09:31:21.634309 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:21.636249 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 236/478] 10.222.0.25 () {60 vars in 1131 bytes} [Fri Oct 28 09:31:21 2022] POST /identity/v3/roles => generated 144 bytes in 209 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:31:21.636389 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f01724b-e80e-4f8d-b299-4e2392dfe12e None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:21.740377 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f01724b-e80e-4f8d-b299-4e2392dfe12e None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:21.777986 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda797cf-6b87-48cc-b890-54d9d13b8c7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:21.778669 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda797cf-6b87-48cc-b890-54d9d13b8c7b None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:21.778958 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda797cf-6b87-48cc-b890-54d9d13b8c7b None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:21.779238 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda797cf-6b87-48cc-b890-54d9d13b8c7b None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:21.791554 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cda797cf-6b87-48cc-b890-54d9d13b8c7b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:31:21.797003 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-cda797cf-6b87-48cc-b890-54d9d13b8c7b None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 09:31:21.797003 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:21.797003 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:21.797003 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:21.797003 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:21.797003 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:21.797003 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:21.797003 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:21.797003 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:21.797003 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:21.797003 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:21.797003 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:21.797003 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:31:21.797003 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:31:21.797003 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:31:21.797003 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:31:21.799199 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:31:21.799199 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:31:21.799199 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:31:21.799199 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:31:21.799199 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:21.799199 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:21.799199 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:21.799199 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:21.799199 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:21.799199 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:21.799199 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:21.799199 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:21.799199 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:21.799199 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:21.799199 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:21.799199 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:21.801053 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:21.801053 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:21.801053 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:21.801053 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:21.801053 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:31:21.801053 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:31:21.801053 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:31:21.801053 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:31:21.801053 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:31:21.801053 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:31:21.801053 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Oct 28 09:31:21.801053 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:21.801053 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 243/479] 10.222.0.25 () {56 vars in 1114 bytes} [Fri Oct 28 09:31:21 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:21.804887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a473fb-cd28-43fd-91d3-f171bf86755e None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:21.819526 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:21.916305 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a473fb-cd28-43fd-91d3-f171bf86755e None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:21.952381 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f8f818-6617-4a72-864f-70a46b87ceec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:21.953010 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f8f818-6617-4a72-864f-70a46b87ceec None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:21.953362 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f8f818-6617-4a72-864f-70a46b87ceec None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:21.953645 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f8f818-6617-4a72-864f-70a46b87ceec None admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:21.954328 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49f8f818-6617-4a72-864f-70a46b87ceec None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:21.954661 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49f8f818-6617-4a72-864f-70a46b87ceec None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:21.956121 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49f8f818-6617-4a72-864f-70a46b87ceec None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:21.967824 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 237/480] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:31:21 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:22.992887 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ce1983-c651-470f-990b-422b489103dc None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:22.993889 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ce1983-c651-470f-990b-422b489103dc None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:22.994366 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ce1983-c651-470f-990b-422b489103dc None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:22.996205 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 244/481] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:23.002025 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11618257-fd0a-473c-ade0-f3f2fb794e76 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:23.002822 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11618257-fd0a-473c-ade0-f3f2fb794e76 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:23.003293 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11618257-fd0a-473c-ade0-f3f2fb794e76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:23.022025 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-11618257-fd0a-473c-ade0-f3f2fb794e76 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:23.023770 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-11618257-fd0a-473c-ade0-f3f2fb794e76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:23.088471 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:23.102514 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 238/482] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:31:23.384451 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ce1983-c651-470f-990b-422b489103dc None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:23.400581 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9dccaf-2dee-44eb-bcfa-3f430cd1114a None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:23.401184 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9dccaf-2dee-44eb-bcfa-3f430cd1114a None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:23.401650 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9dccaf-2dee-44eb-bcfa-3f430cd1114a None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:23.403207 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 239/483] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:23 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 09:31:23.410624 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d25c24-05e9-43da-b22b-629b977c9e52 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:23.411181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d25c24-05e9-43da-b22b-629b977c9e52 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:23.411645 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d25c24-05e9-43da-b22b-629b977c9e52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:23.431856 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-b2d25c24-05e9-43da-b22b-629b977c9e52 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:23.433366 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-b2d25c24-05e9-43da-b22b-629b977c9e52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:23.482176 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:23.558869 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 240/484] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 09:31:23.604423 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ce1983-c651-470f-990b-422b489103dc None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:23.655036 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8cb7ca7-74c2-472e-a9ae-ab2a61abef50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:23.655597 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8cb7ca7-74c2-472e-a9ae-ab2a61abef50 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:23.655878 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8cb7ca7-74c2-472e-a9ae-ab2a61abef50 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:23.656156 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8cb7ca7-74c2-472e-a9ae-ab2a61abef50 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:23.671635 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8cb7ca7-74c2-472e-a9ae-ab2a61abef50 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:31:23.676516 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-b8cb7ca7-74c2-472e-a9ae-ab2a61abef50 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 09:31:23.676516 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:23.676516 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:23.676516 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:23.676516 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:23.676516 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:23.676516 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:23.676516 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:23.676516 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:23.676516 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:23.676516 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:23.676516 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:23.676516 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:31:23.676516 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:31:23.676516 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:31:23.676516 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:31:23.677469 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:31:23.677469 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:31:23.677469 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:31:23.677469 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:31:23.677469 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:23.677469 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:23.677469 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:23.677469 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:23.677469 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:23.677469 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:23.677469 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:23.677469 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:23.677469 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:23.677469 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:23.677469 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:23.677469 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:23.678657 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:23.678657 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:23.678657 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:23.678657 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:23.678657 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:31:23.678657 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:31:23.678657 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:31:23.678657 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:31:23.678657 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:31:23.678657 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:31:23.678657 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 09:31:23.678657 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:23.678657 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 245/485] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Oct 28 09:31:23 2022] GET /identity/v3/roles/service => generated 85 bytes in 294 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:23.687986 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d25c24-05e9-43da-b22b-629b977c9e52 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:23.826685 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d25c24-05e9-43da-b22b-629b977c9e52 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:23.853922 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:23.863184 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d18c9b3-40f9-4674-ba6c-6978a5e7faac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:23.863801 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d18c9b3-40f9-4674-ba6c-6978a5e7faac None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:23.864083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d18c9b3-40f9-4674-ba6c-6978a5e7faac None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:23.864356 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d18c9b3-40f9-4674-ba6c-6978a5e7faac None admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:23.865026 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d18c9b3-40f9-4674-ba6c-6978a5e7faac None admin] RBAC: Adding query filter params (name=service) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:23.865328 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d18c9b3-40f9-4674-ba6c-6978a5e7faac None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:23.866798 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d18c9b3-40f9-4674-ba6c-6978a5e7faac None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:23.875988 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 241/486] 10.222.0.25 () {56 vars in 1103 bytes} [Fri Oct 28 09:31:23 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:23.887498 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8cb7ca7-74c2-472e-a9ae-ab2a61abef50 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:23.936574 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d18c9b3-40f9-4674-ba6c-6978a5e7faac None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:24.017242 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8cb7ca7-74c2-472e-a9ae-ab2a61abef50 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:24.036385 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:24.063136 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d18c9b3-40f9-4674-ba6c-6978a5e7faac None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:24.063654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a798ae00-f29f-4ad6-be4c-ea61e4c5c92c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:24.064171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a798ae00-f29f-4ad6-be4c-ea61e4c5c92c None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:24.064452 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a798ae00-f29f-4ad6-be4c-ea61e4c5c92c None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:24.065006 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a798ae00-f29f-4ad6-be4c-ea61e4c5c92c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:24.080415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a798ae00-f29f-4ad6-be4c-ea61e4c5c92c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:24.082467 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a798ae00-f29f-4ad6-be4c-ea61e4c5c92c None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:24.104968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-a798ae00-f29f-4ad6-be4c-ea61e4c5c92c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:24.104968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:24.104968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:24.104968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:24.104968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:24.104968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:24.104968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:24.104968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:24.104968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:24.104968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:24.104968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:24.104968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:24.104968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:24.104968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:24.104968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:24.104968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:24.105938 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:24.105938 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:24.105938 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:24.105938 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:24.105938 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:24.105938 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:24.105938 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:24.105938 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:24.105938 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:24.105938 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:24.105938 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:24.105938 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:24.105938 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:24.105938 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:24.105938 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:24.105938 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:24.105938 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:24.106989 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:24.106989 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:24.106989 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:24.107167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3695bf6-1391-400a-9a7e-d71712f5cd5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:24.107255 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 246/487] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:24.107319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3695bf6-1391-400a-9a7e-d71712f5cd5b None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:24.107319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3695bf6-1391-400a-9a7e-d71712f5cd5b None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:24.107440 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3695bf6-1391-400a-9a7e-d71712f5cd5b None admin] PATH_INFO: `/v3/roles/service` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:24.107607 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bf40d5-ce62-41a4-8c60-d9316420ea93 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:24.108631 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bf40d5-ce62-41a4-8c60-d9316420ea93 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:24.108631 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bf40d5-ce62-41a4-8c60-d9316420ea93 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:24.109902 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 247/488] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:24.120132 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0835117f-f07c-421f-ab74-b602dd08d78a None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:24.120422 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0835117f-f07c-421f-ab74-b602dd08d78a None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:24.120695 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0835117f-f07c-421f-ab74-b602dd08d78a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:24.128512 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3695bf6-1391-400a-9a7e-d71712f5cd5b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:31:24.134232 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-e3695bf6-1391-400a-9a7e-d71712f5cd5b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 09:31:24.134232 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:24.134232 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:24.134232 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:24.134232 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:24.134232 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:24.134232 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:24.134232 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:24.134232 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:24.134232 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:24.134232 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:24.134232 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:24.134232 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:31:24.134232 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:31:24.134232 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:31:24.134232 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:31:24.136595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:31:24.136595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:31:24.136595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:31:24.136595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:31:24.136595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:24.136595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:24.136595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:24.136595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:24.136595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:24.136595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:24.136595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:24.136595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:24.136595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:24.136595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:24.136595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:24.136595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:24.137818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:24.137818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:24.137818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:24.137818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:24.137818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:31:24.137818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:31:24.137818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:31:24.137818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:31:24.137818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:31:24.137818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:31:24.137818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 09:31:24.137818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:24.137818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 242/489] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Oct 28 09:31:23 2022] GET /identity/v3/roles/service => generated 85 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:31:24.137818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3695bf6-1391-400a-9a7e-d71712f5cd5b None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:24.141356 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-0835117f-f07c-421f-ab74-b602dd08d78a None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:24.142568 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-0835117f-f07c-421f-ab74-b602dd08d78a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:24.204621 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:24.218613 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 248/490] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 09:31:24.219484 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0835117f-f07c-421f-ab74-b602dd08d78a None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:24.313379 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3695bf6-1391-400a-9a7e-d71712f5cd5b None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:24.383947 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff642379-b358-4b90-ae34-870b2d702ca8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:24.384557 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff642379-b358-4b90-ae34-870b2d702ca8 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:24.384834 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff642379-b358-4b90-ae34-870b2d702ca8 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:24.385104 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff642379-b358-4b90-ae34-870b2d702ca8 None admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:24.385750 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff642379-b358-4b90-ae34-870b2d702ca8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:24.386048 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff642379-b358-4b90-ae34-870b2d702ca8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:24.387441 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff642379-b358-4b90-ae34-870b2d702ca8 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:24.409582 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:24.412497 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 243/491] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:24.417301 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff642379-b358-4b90-ae34-870b2d702ca8 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:24.420045 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0835117f-f07c-421f-ab74-b602dd08d78a None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:24.461527 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4eadac3-18f4-438b-ac2a-cc2219f31380 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:24.462057 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4eadac3-18f4-438b-ac2a-cc2219f31380 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:24.462337 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4eadac3-18f4-438b-ac2a-cc2219f31380 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:24.462640 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4eadac3-18f4-438b-ac2a-cc2219f31380 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:24.463303 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4eadac3-18f4-438b-ac2a-cc2219f31380 None admin] RBAC: Adding query filter params (name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:24.463604 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4eadac3-18f4-438b-ac2a-cc2219f31380 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:24.465101 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4eadac3-18f4-438b-ac2a-cc2219f31380 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:24.475060 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 249/492] 10.222.0.25 () {56 vars in 1103 bytes} [Fri Oct 28 09:31:24 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:24.476990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20390891-06e0-4fde-93ec-1ff7701f2e2f None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:24.477278 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20390891-06e0-4fde-93ec-1ff7701f2e2f None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:24.477548 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20390891-06e0-4fde-93ec-1ff7701f2e2f None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:24.478986 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 250/493] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:24.483619 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20390891-06e0-4fde-93ec-1ff7701f2e2f None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:24.553424 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff642379-b358-4b90-ae34-870b2d702ca8 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:24.573694 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:24.609531 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d78f8790-1829-42b9-a08b-4c7e2c1aa24c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:24.610172 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78f8790-1829-42b9-a08b-4c7e2c1aa24c None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:24.610494 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78f8790-1829-42b9-a08b-4c7e2c1aa24c None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:24.611359 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78f8790-1829-42b9-a08b-4c7e2c1aa24c None admin] PATH_INFO: `/v3/users/nova` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:24.620430 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d78f8790-1829-42b9-a08b-4c7e2c1aa24c None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:24.621896 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d78f8790-1829-42b9-a08b-4c7e2c1aa24c None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:24.637887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-d78f8790-1829-42b9-a08b-4c7e2c1aa24c None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 28 09:31:24.637887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:24.637887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:24.637887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:24.637887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:24.637887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:24.637887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:24.637887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:24.637887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:24.637887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:24.637887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:24.637887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:24.637887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 09:31:24.637887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 09:31:24.637887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 09:31:24.637887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 09:31:24.638986 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:24.638986 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:24.638986 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 09:31:24.638986 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:31:24.638986 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 09:31:24.638986 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:31:24.638986 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:24.638986 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:24.638986 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:24.638986 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:24.638986 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:24.638986 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:24.638986 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:24.638986 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:24.638986 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:24.638986 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:24.638986 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:24.638986 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:24.639952 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:24.639952 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:24.639952 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 09:31:24.639952 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 09:31:24.639952 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 09:31:24.639952 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 09:31:24.639952 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 09:31:24.639952 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 09:31:24.639952 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 28 09:31:24.639952 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:24.639952 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 244/494] 10.222.0.25 () {56 vars in 1093 bytes} [Fri Oct 28 09:31:24 2022] GET /identity/v3/users/nova => generated 82 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:24.640608 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9d44c3-aa30-4549-9128-718bab11b6d2 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:24.640845 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9d44c3-aa30-4549-9128-718bab11b6d2 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:24.641124 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9d44c3-aa30-4549-9128-718bab11b6d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:24.648015 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20390891-06e0-4fde-93ec-1ff7701f2e2f None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:24.660024 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-9a9d44c3-aa30-4549-9128-718bab11b6d2 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:24.661229 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-9a9d44c3-aa30-4549-9128-718bab11b6d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:24.693823 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d69cc6e-c2f1-4800-9467-fedda4527099 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:24.694876 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d69cc6e-c2f1-4800-9467-fedda4527099 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:24.695305 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d69cc6e-c2f1-4800-9467-fedda4527099 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:24.695749 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d69cc6e-c2f1-4800-9467-fedda4527099 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:24.707362 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d69cc6e-c2f1-4800-9467-fedda4527099 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:24.709189 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d69cc6e-c2f1-4800-9467-fedda4527099 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:24.726642 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-4d69cc6e-c2f1-4800-9467-fedda4527099 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:24.726642 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:24.726642 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:24.726642 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:24.726642 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:24.726642 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:24.726642 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:24.726642 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:24.726642 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:24.726642 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:24.726642 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:24.726642 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:24.726642 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:24.726642 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:24.726642 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:24.726642 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:24.727711 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:24.727711 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:24.727711 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:24.727711 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:24.727711 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:24.727711 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:24.727711 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:24.727711 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:24.727711 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:24.727711 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:24.727711 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:24.727711 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:24.727711 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:24.727711 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:24.727711 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:24.727711 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:24.727711 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:24.728773 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:24.728773 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:24.728773 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:24.734946 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 251/495] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:24.736159 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d69cc6e-c2f1-4800-9467-fedda4527099 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:24.737577 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 245/496] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:31:24.746537 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9d44c3-aa30-4549-9128-718bab11b6d2 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:24.759111 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:24.849163 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:24.853865 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9d44c3-aa30-4549-9128-718bab11b6d2 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:24.863742 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d69cc6e-c2f1-4800-9467-fedda4527099 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:24.891690 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d6771e-9989-448d-8f7c-67a8dcbfa821 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:24.892332 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d6771e-9989-448d-8f7c-67a8dcbfa821 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:24.892611 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d6771e-9989-448d-8f7c-67a8dcbfa821 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:24.892884 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d6771e-9989-448d-8f7c-67a8dcbfa821 None admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:24.893714 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67d6771e-9989-448d-8f7c-67a8dcbfa821 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:24.894022 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67d6771e-9989-448d-8f7c-67a8dcbfa821 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:24.895645 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67d6771e-9989-448d-8f7c-67a8dcbfa821 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:24.903652 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1623ea43-fd4a-4e3a-9df6-f0ea7e5509d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:24.904610 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1623ea43-fd4a-4e3a-9df6-f0ea7e5509d8 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:24.905066 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1623ea43-fd4a-4e3a-9df6-f0ea7e5509d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:24.905514 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1623ea43-fd4a-4e3a-9df6-f0ea7e5509d8 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:24.917787 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1623ea43-fd4a-4e3a-9df6-f0ea7e5509d8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:31:24.923702 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-1623ea43-fd4a-4e3a-9df6-f0ea7e5509d8 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 09:31:24.923702 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:24.923702 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:24.923702 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:24.923702 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:24.923702 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:24.923702 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:24.923702 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:24.923702 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:24.923702 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:24.923702 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:24.923702 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:24.923702 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:31:24.923702 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:31:24.923702 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:31:24.923702 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:31:24.924593 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:31:24.924593 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:31:24.924593 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:31:24.924593 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:31:24.924593 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:24.924593 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:24.924593 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:24.924593 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:24.924593 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:24.924593 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:24.924593 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:24.924593 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:24.924593 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:24.924593 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:24.924593 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:24.924593 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:24.925385 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:24.925385 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:24.925385 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:24.925385 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:24.925385 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:31:24.925385 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:31:24.925385 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:31:24.925385 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:31:24.925385 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:31:24.925385 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:31:24.925385 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 09:31:24.925385 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:24.933147 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 252/497] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Oct 28 09:31:24 2022] GET /identity/v3/roles/service => generated 85 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:24.934422 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1623ea43-fd4a-4e3a-9df6-f0ea7e5509d8 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:24.942513 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 246/498] 10.222.0.25 () {56 vars in 1133 bytes} [Fri Oct 28 09:31:24 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:24.943474 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d6771e-9989-448d-8f7c-67a8dcbfa821 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:25.013759 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:25.055268 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d6771e-9989-448d-8f7c-67a8dcbfa821 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:25.055812 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1623ea43-fd4a-4e3a-9df6-f0ea7e5509d8 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:25.097539 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1fc4b3e-4a9c-4867-bed2-b967c3d721db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:25.098180 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1fc4b3e-4a9c-4867-bed2-b967c3d721db None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:25.098440 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1fc4b3e-4a9c-4867-bed2-b967c3d721db None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:25.098739 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1fc4b3e-4a9c-4867-bed2-b967c3d721db None admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:25.099380 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1fc4b3e-4a9c-4867-bed2-b967c3d721db None admin] RBAC: Adding query filter params (name=service) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:25.099681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1fc4b3e-4a9c-4867-bed2-b967c3d721db None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:25.101093 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1fc4b3e-4a9c-4867-bed2-b967c3d721db None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:25.102550 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88695778-fa2d-4ecb-9f74-2089aecb3a72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:25.103250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88695778-fa2d-4ecb-9f74-2089aecb3a72 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:25.103657 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88695778-fa2d-4ecb-9f74-2089aecb3a72 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:25.103938 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88695778-fa2d-4ecb-9f74-2089aecb3a72 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:25.104586 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88695778-fa2d-4ecb-9f74-2089aecb3a72 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:25.104894 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88695778-fa2d-4ecb-9f74-2089aecb3a72 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:25.106263 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88695778-fa2d-4ecb-9f74-2089aecb3a72 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:25.117715 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 247/499] 10.222.0.25 () {56 vars in 1103 bytes} [Fri Oct 28 09:31:24 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:25.117715 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1fc4b3e-4a9c-4867-bed2-b967c3d721db None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:25.132001 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 253/500] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:25.132001 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88695778-fa2d-4ecb-9f74-2089aecb3a72 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:25.142117 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:25.230614 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1fc4b3e-4a9c-4867-bed2-b967c3d721db None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:25.241555 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88695778-fa2d-4ecb-9f74-2089aecb3a72 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:25.269311 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e52649-49bc-4dc1-9d7d-ca68f5288338 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:25.269855 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e52649-49bc-4dc1-9d7d-ca68f5288338 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:25.270130 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e52649-49bc-4dc1-9d7d-ca68f5288338 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:25.270219 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:25.270429 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e52649-49bc-4dc1-9d7d-ca68f5288338 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:25.278623 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2229983c-b6cc-4ff7-838b-d2e158810f0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:25.279247 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2229983c-b6cc-4ff7-838b-d2e158810f0e None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:25.279531 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2229983c-b6cc-4ff7-838b-d2e158810f0e None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:25.279802 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2229983c-b6cc-4ff7-838b-d2e158810f0e None admin] PATH_INFO: `/v3/roles/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:25.291883 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2229983c-b6cc-4ff7-838b-d2e158810f0e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:31:25.296688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-2229983c-b6cc-4ff7-838b-d2e158810f0e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 09:31:25.296688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:25.296688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:25.296688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:25.296688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:25.296688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:25.296688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:25.296688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:25.296688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:25.296688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:25.296688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:25.296688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:25.296688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:31:25.296688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:31:25.296688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:31:25.296688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:31:25.298103 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:31:25.298103 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:31:25.298103 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:31:25.298103 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:31:25.298103 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:25.298103 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:25.298103 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:25.298103 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:25.298103 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:25.298103 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:25.298103 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:25.298103 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:25.298103 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:25.298103 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:25.298103 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:25.298103 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:25.299468 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:25.299468 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:25.299468 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:25.299468 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:25.299468 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:31:25.299468 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:31:25.299468 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:31:25.299468 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:31:25.299468 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:31:25.299468 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:31:25.299468 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 09:31:25.299468 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:25.299468 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 254/501] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Oct 28 09:31:25 2022] GET /identity/v3/roles/service => generated 85 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:25.299468 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2229983c-b6cc-4ff7-838b-d2e158810f0e None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:25.352101 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e52649-49bc-4dc1-9d7d-ca68f5288338 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:25.353554 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e52649-49bc-4dc1-9d7d-ca68f5288338 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:25.373928 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:25.409908 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2229983c-b6cc-4ff7-838b-d2e158810f0e None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:25.445859 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bfb3827-9fd6-4f29-81c5-afc74a506dc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:25.446571 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bfb3827-9fd6-4f29-81c5-afc74a506dc3 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:25.446812 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bfb3827-9fd6-4f29-81c5-afc74a506dc3 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:25.447073 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bfb3827-9fd6-4f29-81c5-afc74a506dc3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:25.457931 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bfb3827-9fd6-4f29-81c5-afc74a506dc3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:25.459401 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bfb3827-9fd6-4f29-81c5-afc74a506dc3 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:25.464605 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 248/502] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:31:25 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 09:31:25.465246 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e52649-49bc-4dc1-9d7d-ca68f5288338 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:25.478152 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-8bfb3827-9fd6-4f29-81c5-afc74a506dc3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:25.478152 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:25.478152 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:25.478152 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:25.478152 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:25.478152 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:25.478152 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:25.478152 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:25.478152 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:25.478152 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:25.478152 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:25.478152 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:25.478152 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:25.478152 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:25.478152 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:25.478152 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:25.479195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:25.479195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:25.479195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:25.479195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:25.479195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:25.479195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:25.479195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:25.479195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:25.479195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:25.479195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:25.479195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:25.479195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:25.479195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:25.479195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:25.479195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:25.479195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:25.479195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:25.480045 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:25.480045 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:25.480045 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:25.480045 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 255/503] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:25.481087 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351983c5-8a40-48f6-b247-3e9ba9d3d7ed None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:25.481378 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351983c5-8a40-48f6-b247-3e9ba9d3d7ed None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:25.481653 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351983c5-8a40-48f6-b247-3e9ba9d3d7ed None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:25.485217 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 256/504] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:25.485217 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351983c5-8a40-48f6-b247-3e9ba9d3d7ed None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:25.536294 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:25.584701 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e52649-49bc-4dc1-9d7d-ca68f5288338 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:25.602978 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351983c5-8a40-48f6-b247-3e9ba9d3d7ed None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:25.604569 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:25.624693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32beffd7-fa24-4165-9a68-8ce88cb3d00d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:25.625324 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32beffd7-fa24-4165-9a68-8ce88cb3d00d None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:25.625606 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32beffd7-fa24-4165-9a68-8ce88cb3d00d None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:25.625882 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32beffd7-fa24-4165-9a68-8ce88cb3d00d None admin] PATH_INFO: `/v3/users/glance` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:25.634629 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32beffd7-fa24-4165-9a68-8ce88cb3d00d None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:25.636085 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32beffd7-fa24-4165-9a68-8ce88cb3d00d None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:25.639633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff5bee01-7e8e-4105-b687-4b9a7530ead7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:25.640231 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff5bee01-7e8e-4105-b687-4b9a7530ead7 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:25.640506 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff5bee01-7e8e-4105-b687-4b9a7530ead7 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:25.640775 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff5bee01-7e8e-4105-b687-4b9a7530ead7 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:25.641437 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff5bee01-7e8e-4105-b687-4b9a7530ead7 None admin] RBAC: Adding query filter params (name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:25.641734 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff5bee01-7e8e-4105-b687-4b9a7530ead7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:25.643120 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff5bee01-7e8e-4105-b687-4b9a7530ead7 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:25.648495 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-32beffd7-fa24-4165-9a68-8ce88cb3d00d None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 28 09:31:25.648495 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:25.648495 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:25.648495 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:25.648495 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:25.648495 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:25.648495 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:25.648495 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:25.648495 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:25.648495 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:25.648495 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:25.648495 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:25.648495 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 09:31:25.648495 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 09:31:25.648495 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 09:31:25.648495 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 09:31:25.649595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:25.649595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:25.649595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 09:31:25.649595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:31:25.649595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 09:31:25.649595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:31:25.649595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:25.649595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:25.649595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:25.649595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:25.649595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:25.649595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:25.649595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:25.649595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:25.649595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:25.649595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:25.649595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:25.649595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:25.650985 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:25.650985 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:25.650985 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 09:31:25.650985 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 09:31:25.650985 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 09:31:25.650985 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 09:31:25.650985 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 09:31:25.650985 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 09:31:25.650985 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 28 09:31:25.650985 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:25.650985 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 249/505] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:31:25 2022] GET /identity/v3/users/glance => generated 84 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:25.650985 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32beffd7-fa24-4165-9a68-8ce88cb3d00d None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:25.662921 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 257/506] 10.222.0.25 () {56 vars in 1103 bytes} [Fri Oct 28 09:31:25 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:25.662921 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff5bee01-7e8e-4105-b687-4b9a7530ead7 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:25.766205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32beffd7-fa24-4165-9a68-8ce88cb3d00d None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:25.767678 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff5bee01-7e8e-4105-b687-4b9a7530ead7 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:25.769213 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:25.802698 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28826d59-dc3c-4935-ad26-644a7b7c1a83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:25.803378 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28826d59-dc3c-4935-ad26-644a7b7c1a83 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:25.803763 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28826d59-dc3c-4935-ad26-644a7b7c1a83 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:25.804614 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28826d59-dc3c-4935-ad26-644a7b7c1a83 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:25.805551 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0332b79-b50e-415c-8a51-aea9a8ddb1d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:25.806161 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0332b79-b50e-415c-8a51-aea9a8ddb1d0 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:25.806462 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0332b79-b50e-415c-8a51-aea9a8ddb1d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:25.806740 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0332b79-b50e-415c-8a51-aea9a8ddb1d0 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:25.807386 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0332b79-b50e-415c-8a51-aea9a8ddb1d0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:25.807687 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0332b79-b50e-415c-8a51-aea9a8ddb1d0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:25.809061 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0332b79-b50e-415c-8a51-aea9a8ddb1d0 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:25.817969 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28826d59-dc3c-4935-ad26-644a7b7c1a83 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:25.819486 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28826d59-dc3c-4935-ad26-644a7b7c1a83 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:25.822858 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 258/507] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:25.824091 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414fc942-e64b-4fbd-9f24-ac0e813fd099 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:25.824380 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414fc942-e64b-4fbd-9f24-ac0e813fd099 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:25.824659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414fc942-e64b-4fbd-9f24-ac0e813fd099 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:25.835887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-28826d59-dc3c-4935-ad26-644a7b7c1a83 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:25.835887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:25.835887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:25.835887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:25.835887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:25.835887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:25.835887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:25.835887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:25.835887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:25.835887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:25.835887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:25.835887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:25.835887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:25.835887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:25.835887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:25.835887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:25.837381 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:25.837381 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:25.837381 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:25.837381 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:25.837381 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:25.837381 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:25.837381 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:25.837381 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:25.837381 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:25.837381 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:25.837381 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:25.837381 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:25.837381 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:25.837381 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:25.837381 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:25.837381 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:25.837381 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:25.838375 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:25.838375 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:25.838375 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:25.838375 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 250/508] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:25.838375 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28826d59-dc3c-4935-ad26-644a7b7c1a83 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:25.846283 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-414fc942-e64b-4fbd-9f24-ac0e813fd099 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:25.847520 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-414fc942-e64b-4fbd-9f24-ac0e813fd099 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:25.875878 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:25.917602 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 259/509] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:31:25.918765 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-414fc942-e64b-4fbd-9f24-ac0e813fd099 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:25.955300 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28826d59-dc3c-4935-ad26-644a7b7c1a83 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:25.995784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-524804dd-7b31-47ee-a24f-9ec58aa05bdb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:25.996510 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524804dd-7b31-47ee-a24f-9ec58aa05bdb None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:25.996795 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524804dd-7b31-47ee-a24f-9ec58aa05bdb None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:25.997067 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524804dd-7b31-47ee-a24f-9ec58aa05bdb None admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:25.997799 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-524804dd-7b31-47ee-a24f-9ec58aa05bdb None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:25.998102 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-524804dd-7b31-47ee-a24f-9ec58aa05bdb None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:25.999742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-524804dd-7b31-47ee-a24f-9ec58aa05bdb None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:26.025633 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 251/510] 10.222.0.25 () {56 vars in 1137 bytes} [Fri Oct 28 09:31:25 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:26.025633 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-524804dd-7b31-47ee-a24f-9ec58aa05bdb None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:26.038008 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:26.053860 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-414fc942-e64b-4fbd-9f24-ac0e813fd099 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:26.096645 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d56c7b-e5ab-49ad-b427-bbe90a0a0f34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:26.097274 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d56c7b-e5ab-49ad-b427-bbe90a0a0f34 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:26.097548 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d56c7b-e5ab-49ad-b427-bbe90a0a0f34 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:26.097816 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d56c7b-e5ab-49ad-b427-bbe90a0a0f34 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:26.113419 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d56c7b-e5ab-49ad-b427-bbe90a0a0f34 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:26.114299 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6d56c7b-e5ab-49ad-b427-bbe90a0a0f34 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:26.123725 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:26.132803 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-d6d56c7b-e5ab-49ad-b427-bbe90a0a0f34 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:26.132803 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:26.132803 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:26.132803 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:26.132803 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:26.132803 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:26.132803 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:26.132803 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:26.132803 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:26.132803 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:26.132803 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:26.132803 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:26.132803 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:26.132803 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:26.132803 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:26.132803 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:26.133802 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:26.133802 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:26.133802 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:26.133802 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:26.133802 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:26.133802 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:26.133802 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:26.133802 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:26.133802 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:26.133802 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:26.133802 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:26.133802 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:26.133802 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:26.133802 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:26.133802 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:26.133802 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:26.133802 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:26.134688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:26.134688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:26.134688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:26.136079 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 260/511] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:26.137093 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d56c7b-e5ab-49ad-b427-bbe90a0a0f34 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:26.143926 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-524804dd-7b31-47ee-a24f-9ec58aa05bdb None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:26.184329 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5dd1ca-83c8-48e0-9770-0f9002e064f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:26.184981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5dd1ca-83c8-48e0-9770-0f9002e064f3 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:26.185253 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5dd1ca-83c8-48e0-9770-0f9002e064f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:26.185537 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5dd1ca-83c8-48e0-9770-0f9002e064f3 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:26.195433 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d5dd1ca-83c8-48e0-9770-0f9002e064f3 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:26.197052 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d5dd1ca-83c8-48e0-9770-0f9002e064f3 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:26.208948 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-5d5dd1ca-83c8-48e0-9770-0f9002e064f3 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 28 09:31:26.208948 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:26.208948 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:26.208948 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:26.208948 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:26.208948 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:26.208948 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:26.208948 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:26.208948 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:26.208948 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:26.208948 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:26.208948 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:26.208948 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 09:31:26.208948 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 09:31:26.208948 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 09:31:26.208948 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 09:31:26.209981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:26.209981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:26.209981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 09:31:26.209981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:31:26.209981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 09:31:26.209981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:31:26.209981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:26.209981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:26.209981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:26.209981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:26.209981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:26.209981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:26.209981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:26.209981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:26.209981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:26.209981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:26.209981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:26.209981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:26.211625 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:26.211625 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:26.211625 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 09:31:26.211625 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 09:31:26.211625 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 09:31:26.211625 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 09:31:26.211625 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 09:31:26.211625 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 09:31:26.211625 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 28 09:31:26.211625 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:26.211625 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 252/512] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:31:26 2022] GET /identity/v3/users/cinder => generated 84 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:26.211625 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5dd1ca-83c8-48e0-9770-0f9002e064f3 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:26.253476 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d56c7b-e5ab-49ad-b427-bbe90a0a0f34 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:26.291054 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54aac19-2938-4824-b145-249334d14689 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:26.291665 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54aac19-2938-4824-b145-249334d14689 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:26.291944 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54aac19-2938-4824-b145-249334d14689 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:26.292218 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54aac19-2938-4824-b145-249334d14689 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:26.292865 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d54aac19-2938-4824-b145-249334d14689 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:26.293171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d54aac19-2938-4824-b145-249334d14689 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:26.294572 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d54aac19-2938-4824-b145-249334d14689 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:26.307438 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 261/513] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:26.308198 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54aac19-2938-4824-b145-249334d14689 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:26.311690 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:26.323151 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5dd1ca-83c8-48e0-9770-0f9002e064f3 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:26.357166 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b89c0f-97ba-4b06-bd34-4566cc85af7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:26.357776 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b89c0f-97ba-4b06-bd34-4566cc85af7a None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:26.358058 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b89c0f-97ba-4b06-bd34-4566cc85af7a None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:26.358327 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b89c0f-97ba-4b06-bd34-4566cc85af7a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:26.377202 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:26.418913 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54aac19-2938-4824-b145-249334d14689 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:26.432920 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64b89c0f-97ba-4b06-bd34-4566cc85af7a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:26.434546 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64b89c0f-97ba-4b06-bd34-4566cc85af7a None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:26.457458 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40641ca8-16cb-4fa2-9b61-0cbae015fa77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:26.458069 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40641ca8-16cb-4fa2-9b61-0cbae015fa77 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:26.458349 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40641ca8-16cb-4fa2-9b61-0cbae015fa77 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:26.458650 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40641ca8-16cb-4fa2-9b61-0cbae015fa77 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:26.459288 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40641ca8-16cb-4fa2-9b61-0cbae015fa77 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:26.459599 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40641ca8-16cb-4fa2-9b61-0cbae015fa77 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:26.461000 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40641ca8-16cb-4fa2-9b61-0cbae015fa77 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:26.475538 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 262/514] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:26.475538 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40641ca8-16cb-4fa2-9b61-0cbae015fa77 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:26.544027 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 253/515] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:31:26 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 09:31:26.544987 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b89c0f-97ba-4b06-bd34-4566cc85af7a None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:26.548273 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:26.591754 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40641ca8-16cb-4fa2-9b61-0cbae015fa77 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:26.604491 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:26.629988 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ddca1a-e5e9-4972-9899-41925fed1ab9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:26.630646 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ddca1a-e5e9-4972-9899-41925fed1ab9 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:26.630923 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ddca1a-e5e9-4972-9899-41925fed1ab9 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:26.631193 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ddca1a-e5e9-4972-9899-41925fed1ab9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:26.643524 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ddca1a-e5e9-4972-9899-41925fed1ab9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:26.644967 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ddca1a-e5e9-4972-9899-41925fed1ab9 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:26.658683 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b89c0f-97ba-4b06-bd34-4566cc85af7a None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:26.659274 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-21ddca1a-e5e9-4972-9899-41925fed1ab9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:26.659274 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:26.659274 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:26.659274 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:26.659274 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:26.659274 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:26.659274 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:26.659274 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:26.659274 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:26.659274 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:26.659274 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:26.659274 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:26.659274 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:26.659274 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:26.659274 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:26.659274 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:26.660171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:26.660171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:26.660171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:26.660171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:26.660171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:26.660171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:26.660171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:26.660171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:26.660171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:26.660171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:26.660171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:26.660171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:26.660171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:26.660171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:26.660171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:26.660171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:26.660171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:26.661065 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:26.661065 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:26.661065 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:26.661065 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 263/516] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:26.661276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ddca1a-e5e9-4972-9899-41925fed1ab9 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:26.695560 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb088f6-0e76-4384-9c81-74728c23cd2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:26.696182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb088f6-0e76-4384-9c81-74728c23cd2b None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:26.696462 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb088f6-0e76-4384-9c81-74728c23cd2b None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:26.696730 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb088f6-0e76-4384-9c81-74728c23cd2b None admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:26.697449 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beb088f6-0e76-4384-9c81-74728c23cd2b None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:26.697751 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beb088f6-0e76-4384-9c81-74728c23cd2b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:26.699219 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beb088f6-0e76-4384-9c81-74728c23cd2b None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:26.717346 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 254/517] 10.222.0.25 () {56 vars in 1137 bytes} [Fri Oct 28 09:31:26 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:26.718181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb088f6-0e76-4384-9c81-74728c23cd2b None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:26.777306 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ddca1a-e5e9-4972-9899-41925fed1ab9 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:26.796665 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:26.815171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35f32d1-e390-45df-8371-05651699d6f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:26.815800 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35f32d1-e390-45df-8371-05651699d6f2 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:26.816077 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35f32d1-e390-45df-8371-05651699d6f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:26.816348 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35f32d1-e390-45df-8371-05651699d6f2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:26.823591 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a35f32d1-e390-45df-8371-05651699d6f2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:26.824999 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a35f32d1-e390-45df-8371-05651699d6f2 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:26.830560 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb088f6-0e76-4384-9c81-74728c23cd2b None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:26.839130 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:26.841132 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-a35f32d1-e390-45df-8371-05651699d6f2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:31:26.841132 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:26.841132 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:26.841132 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:26.841132 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:26.841132 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:26.841132 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:26.841132 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:26.841132 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:26.841132 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:26.841132 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:26.841132 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:26.841132 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 09:31:26.841132 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 09:31:26.841132 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 09:31:26.842177 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 09:31:26.842177 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:26.842177 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:26.842177 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:26.842177 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:26.842177 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:26.842177 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:26.842177 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:26.842177 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:26.842177 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:26.842177 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:26.842177 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:26.842177 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:26.842177 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:26.842177 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:26.842177 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:26.842177 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:26.843116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 09:31:26.843116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 09:31:26.843116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 09:31:26.843116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 09:31:26.843116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 09:31:26.843116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 09:31:26.843116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:31:26.843116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:26.843116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 264/518] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:31:26 2022] GET /identity/v3/projects/service => generated 88 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:26.843577 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35f32d1-e390-45df-8371-05651699d6f2 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:26.878409 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0841e5d4-adf9-4673-ad0b-ca0bdae0088e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:26.879058 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0841e5d4-adf9-4673-ad0b-ca0bdae0088e None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:26.879330 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0841e5d4-adf9-4673-ad0b-ca0bdae0088e None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:26.879609 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0841e5d4-adf9-4673-ad0b-ca0bdae0088e None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:26.887723 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0841e5d4-adf9-4673-ad0b-ca0bdae0088e None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:26.889175 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0841e5d4-adf9-4673-ad0b-ca0bdae0088e None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:26.901784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-0841e5d4-adf9-4673-ad0b-ca0bdae0088e None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 28 09:31:26.901784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:26.901784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:26.901784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:26.901784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:26.901784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:26.901784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:26.901784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:26.901784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:26.901784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:26.901784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:26.901784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:26.901784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 09:31:26.901784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 09:31:26.901784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 09:31:26.901784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 09:31:26.902927 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:26.902927 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:26.902927 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 09:31:26.902927 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:31:26.902927 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 09:31:26.902927 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:31:26.902927 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:26.902927 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:26.902927 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:26.902927 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:26.902927 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:26.902927 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:26.902927 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:26.902927 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:26.902927 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:26.902927 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:26.902927 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:26.902927 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:26.904339 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:26.904339 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:26.904339 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 09:31:26.904339 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 09:31:26.904339 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 09:31:26.904339 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 09:31:26.904339 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 09:31:26.904339 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 09:31:26.904339 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 28 09:31:26.904339 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:26.904339 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 255/519] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Oct 28 09:31:26 2022] GET /identity/v3/users/neutron => generated 85 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:26.904339 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0841e5d4-adf9-4673-ad0b-ca0bdae0088e None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:26.956331 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35f32d1-e390-45df-8371-05651699d6f2 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:26.994895 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a76471-eb8a-4735-93f9-f39ac805ddf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:26.995528 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a76471-eb8a-4735-93f9-f39ac805ddf2 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:26.995809 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a76471-eb8a-4735-93f9-f39ac805ddf2 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:26.996089 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21a76471-eb8a-4735-93f9-f39ac805ddf2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:27.007749 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a76471-eb8a-4735-93f9-f39ac805ddf2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:27.009050 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0841e5d4-adf9-4673-ad0b-ca0bdae0088e None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:27.009182 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21a76471-eb8a-4735-93f9-f39ac805ddf2 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:27.024224 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-21a76471-eb8a-4735-93f9-f39ac805ddf2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:27.024224 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:27.024224 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:27.024224 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:27.024224 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:27.024224 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:27.024224 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:27.024224 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:27.024224 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:27.024224 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:27.024224 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:27.024224 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:27.024224 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:27.024224 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:27.024224 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:27.024224 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:27.025212 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:27.025212 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:27.025212 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:27.025212 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:27.025212 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:27.025212 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:27.025212 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:27.025212 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:27.025212 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:27.025212 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:27.025212 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:27.025212 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:27.025212 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:27.025212 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:27.025212 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:27.025212 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:27.025212 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:27.026053 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:27.026053 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:27.026053 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:27.026053 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 265/520] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:27.026455 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a76471-eb8a-4735-93f9-f39ac805ddf2 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:27.049308 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad3a43c5-df71-49ac-a52f-78685ebd47eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:27.050030 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3a43c5-df71-49ac-a52f-78685ebd47eb None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:27.050411 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3a43c5-df71-49ac-a52f-78685ebd47eb None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:27.050795 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3a43c5-df71-49ac-a52f-78685ebd47eb None admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:27.051773 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad3a43c5-df71-49ac-a52f-78685ebd47eb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:27.052173 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad3a43c5-df71-49ac-a52f-78685ebd47eb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:27.053821 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad3a43c5-df71-49ac-a52f-78685ebd47eb None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:27.058924 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:27.066088 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 256/521] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:27.066999 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad3a43c5-df71-49ac-a52f-78685ebd47eb None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:27.097126 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:27.144433 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21a76471-eb8a-4735-93f9-f39ac805ddf2 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:27.181119 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad3a43c5-df71-49ac-a52f-78685ebd47eb None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:27.182243 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8319047-260a-4e30-a858-992040876446 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:27.182376 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8319047-260a-4e30-a858-992040876446 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:27.182663 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8319047-260a-4e30-a858-992040876446 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:27.182934 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8319047-260a-4e30-a858-992040876446 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:27.219290 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7878d338-a530-4559-8905-cb60240255f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:27.219898 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7878d338-a530-4559-8905-cb60240255f3 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:27.220176 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7878d338-a530-4559-8905-cb60240255f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:27.220451 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7878d338-a530-4559-8905-cb60240255f3 None admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:27.221143 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7878d338-a530-4559-8905-cb60240255f3 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:27.221444 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7878d338-a530-4559-8905-cb60240255f3 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:27.222917 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7878d338-a530-4559-8905-cb60240255f3 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:27.240659 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 257/522] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Oct 28 09:31:27 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:27.242025 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7878d338-a530-4559-8905-cb60240255f3 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:27.263865 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8319047-260a-4e30-a858-992040876446 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:27.265447 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8319047-260a-4e30-a858-992040876446 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:27.295312 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:27.341901 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:27.358113 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7878d338-a530-4559-8905-cb60240255f3 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:27.379828 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 266/523] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:31:27 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 354 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 09:31:27.380694 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8319047-260a-4e30-a858-992040876446 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:27.398431 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e60e375-f173-4599-afbe-a66906d45bf8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:27.399033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e60e375-f173-4599-afbe-a66906d45bf8 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:27.399311 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e60e375-f173-4599-afbe-a66906d45bf8 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:27.399579 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e60e375-f173-4599-afbe-a66906d45bf8 None admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:27.400413 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e60e375-f173-4599-afbe-a66906d45bf8 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:27.400714 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e60e375-f173-4599-afbe-a66906d45bf8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:27.402232 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e60e375-f173-4599-afbe-a66906d45bf8 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:27.421259 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 258/524] 10.222.0.25 () {56 vars in 1139 bytes} [Fri Oct 28 09:31:27 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:27.422082 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e60e375-f173-4599-afbe-a66906d45bf8 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:27.493202 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8319047-260a-4e30-a858-992040876446 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:27.520736 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:27.528993 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c85d73-a941-4e75-8e1c-01e03834fd0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:27.529613 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c85d73-a941-4e75-8e1c-01e03834fd0d None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:27.529894 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c85d73-a941-4e75-8e1c-01e03834fd0d None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:27.529998 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e60e375-f173-4599-afbe-a66906d45bf8 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:27.530164 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c85d73-a941-4e75-8e1c-01e03834fd0d None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:27.530830 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9c85d73-a941-4e75-8e1c-01e03834fd0d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:27.531129 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9c85d73-a941-4e75-8e1c-01e03834fd0d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:27.532749 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9c85d73-a941-4e75-8e1c-01e03834fd0d None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:27.546819 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 267/525] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:27.547673 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c85d73-a941-4e75-8e1c-01e03834fd0d None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:27.573329 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2d7fd8-1cbd-42ef-9207-ca9a04c38675 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:27.573953 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2d7fd8-1cbd-42ef-9207-ca9a04c38675 None admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:27.574229 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2d7fd8-1cbd-42ef-9207-ca9a04c38675 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:27.574537 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2d7fd8-1cbd-42ef-9207-ca9a04c38675 None admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:27.575194 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f2d7fd8-1cbd-42ef-9207-ca9a04c38675 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:27.576635 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f2d7fd8-1cbd-42ef-9207-ca9a04c38675 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:27.594554 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:27.610521 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 259/526] 10.222.0.25 () {60 vars in 1132 bytes} [Fri Oct 28 09:31:27 2022] POST /identity/v3/users => generated 247 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:31:27.610521 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2d7fd8-1cbd-42ef-9207-ca9a04c38675 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:27.691905 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c85d73-a941-4e75-8e1c-01e03834fd0d None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:27.727644 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2d7fd8-1cbd-42ef-9207-ca9a04c38675 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:27.735807 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4855fa05-df66-4289-b318-c7b9ce7cd016 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:27.736442 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4855fa05-df66-4289-b318-c7b9ce7cd016 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:27.736725 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4855fa05-df66-4289-b318-c7b9ce7cd016 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:27.737001 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4855fa05-df66-4289-b318-c7b9ce7cd016 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:27.737753 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4855fa05-df66-4289-b318-c7b9ce7cd016 None admin] RBAC: Adding query filter params (role.id=e5d8d090857a4a018d0ec3c79ece6472, scope.project.id=455d1c826b974bc38497d18a516867c7, user.id=688ceb201d304c2da5373c3cf36c2d79) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:27.738064 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4855fa05-df66-4289-b318-c7b9ce7cd016 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:27.742300 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4855fa05-df66-4289-b318-c7b9ce7cd016 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:27.746377 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 268/527] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:31:27 2022] GET /identity/v3/role_assignments?user.id=688ceb201d304c2da5373c3cf36c2d79&scope.project.id=455d1c826b974bc38497d18a516867c7&role.id=e5d8d090857a4a018d0ec3c79ece6472 => generated 259 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:27.747287 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4855fa05-df66-4289-b318-c7b9ce7cd016 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:27.792608 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19520e9-c173-4ae8-a5b0-c0d4355fc47e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:27.793287 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19520e9-c173-4ae8-a5b0-c0d4355fc47e None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:27.793567 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19520e9-c173-4ae8-a5b0-c0d4355fc47e None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:27.793837 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f19520e9-c173-4ae8-a5b0-c0d4355fc47e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:27.814450 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f19520e9-c173-4ae8-a5b0-c0d4355fc47e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:27.814870 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f19520e9-c173-4ae8-a5b0-c0d4355fc47e None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:27.829262 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:27.837111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-f19520e9-c173-4ae8-a5b0-c0d4355fc47e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:27.837111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:27.837111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:27.837111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:27.837111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:27.837111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:27.837111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:27.837111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:27.837111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:27.837111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:27.837111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:27.837111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:27.837111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:27.837111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:27.837111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:27.837111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:27.838111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:27.838111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:27.838111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:27.838111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:27.838111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:27.838111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:27.838111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:27.838111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:27.838111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:27.838111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:27.838111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:27.838111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:27.838111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:27.838111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:27.838111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:27.838111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:27.838111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:27.839030 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:27.839030 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:27.839030 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:27.839030 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 260/528] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:27.839259 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19520e9-c173-4ae8-a5b0-c0d4355fc47e None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:27.910284 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4855fa05-df66-4289-b318-c7b9ce7cd016 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:27.922449 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:27.968655 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a57306c-0176-402c-9132-591a9684fada None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:27.969287 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a57306c-0176-402c-9132-591a9684fada None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:27.969568 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a57306c-0176-402c-9132-591a9684fada None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:27.969843 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a57306c-0176-402c-9132-591a9684fada None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:27.992269 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f19520e9-c173-4ae8-a5b0-c0d4355fc47e None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:28.037664 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508c8954-93c7-41bc-9565-d82291a0f2d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:28.038739 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508c8954-93c7-41bc-9565-d82291a0f2d4 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:28.039225 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508c8954-93c7-41bc-9565-d82291a0f2d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:28.039709 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508c8954-93c7-41bc-9565-d82291a0f2d4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:28.047363 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-508c8954-93c7-41bc-9565-d82291a0f2d4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:28.049430 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-508c8954-93c7-41bc-9565-d82291a0f2d4 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:28.062775 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a57306c-0176-402c-9132-591a9684fada None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:28.063360 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-508c8954-93c7-41bc-9565-d82291a0f2d4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:31:28.063360 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:28.063360 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:28.063360 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:28.063360 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:28.063360 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:28.063360 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:28.063360 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:28.063360 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:28.063360 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:28.063360 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:28.063360 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:28.063360 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 09:31:28.063360 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 09:31:28.063360 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 09:31:28.064167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 09:31:28.064167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:28.064167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:28.064167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:28.064167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:28.064167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:28.064167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:28.064167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:28.064167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:28.064167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:28.064167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:28.064167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:28.064167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:28.064167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:28.064167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:28.064167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:28.064167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:28.064987 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 09:31:28.064987 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 09:31:28.064987 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 09:31:28.064987 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 09:31:28.064987 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 09:31:28.064987 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 09:31:28.064987 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:31:28.064987 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:28.065384 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a57306c-0176-402c-9132-591a9684fada None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:28.067480 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 261/529] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:31:27 2022] GET /identity/v3/projects/service => generated 88 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:28.068548 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508c8954-93c7-41bc-9565-d82291a0f2d4 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:28.129098 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:28.187094 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508c8954-93c7-41bc-9565-d82291a0f2d4 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:28.195208 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 269/530] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:31:27 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 448 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 09:31:28.196689 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a57306c-0176-402c-9132-591a9684fada None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:28.212073 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:28.241172 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3984339f-9839-40e2-84cb-510273c801d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:28.241618 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3984339f-9839-40e2-84cb-510273c801d4 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:28.241883 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3984339f-9839-40e2-84cb-510273c801d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:28.242154 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3984339f-9839-40e2-84cb-510273c801d4 None admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:28.242843 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3984339f-9839-40e2-84cb-510273c801d4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:28.243150 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3984339f-9839-40e2-84cb-510273c801d4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:28.244551 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3984339f-9839-40e2-84cb-510273c801d4 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:28.259999 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 262/531] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:28.259999 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3984339f-9839-40e2-84cb-510273c801d4 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:28.316705 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a57306c-0176-402c-9132-591a9684fada None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:28.356261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8903ea22-b35a-4dc5-977e-6083e360f43d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:28.357242 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8903ea22-b35a-4dc5-977e-6083e360f43d None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:28.357717 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8903ea22-b35a-4dc5-977e-6083e360f43d None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:28.358274 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8903ea22-b35a-4dc5-977e-6083e360f43d None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:28.359195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8903ea22-b35a-4dc5-977e-6083e360f43d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:28.359717 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8903ea22-b35a-4dc5-977e-6083e360f43d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:28.361570 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8903ea22-b35a-4dc5-977e-6083e360f43d None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:28.370981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3984339f-9839-40e2-84cb-510273c801d4 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:28.381808 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 270/532] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Oct 28 09:31:28 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:28.383694 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8903ea22-b35a-4dc5-977e-6083e360f43d None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:28.400227 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:28.411850 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9775ee6b-0c59-44b6-9ce2-ca308216a56b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:28.412478 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9775ee6b-0c59-44b6-9ce2-ca308216a56b None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:28.412759 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9775ee6b-0c59-44b6-9ce2-ca308216a56b None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:28.413036 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9775ee6b-0c59-44b6-9ce2-ca308216a56b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:28.425776 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9775ee6b-0c59-44b6-9ce2-ca308216a56b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:28.427381 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9775ee6b-0c59-44b6-9ce2-ca308216a56b None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:28.444296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-9775ee6b-0c59-44b6-9ce2-ca308216a56b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:28.444296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:28.444296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:28.444296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:28.444296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:28.444296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:28.444296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:28.444296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:28.444296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:28.444296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:28.444296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:28.444296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:28.444296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:28.444296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:28.444296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:28.444296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:28.445742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:28.445742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:28.445742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:28.445742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:28.445742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:28.445742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:28.445742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:28.445742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:28.445742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:28.445742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:28.445742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:28.445742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:28.445742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:28.445742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:28.445742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:28.445742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:28.445742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:28.446599 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:28.446599 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:28.446599 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:28.446599 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 263/533] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:28.446822 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9775ee6b-0c59-44b6-9ce2-ca308216a56b None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:28.485626 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:28.508972 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8903ea22-b35a-4dc5-977e-6083e360f43d None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:28.548180 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74a5cdf-7d5f-4992-aceb-be966c15bcd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:28.549259 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74a5cdf-7d5f-4992-aceb-be966c15bcd4 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:28.549809 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74a5cdf-7d5f-4992-aceb-be966c15bcd4 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:28.550272 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74a5cdf-7d5f-4992-aceb-be966c15bcd4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:28.560942 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f74a5cdf-7d5f-4992-aceb-be966c15bcd4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:28.562945 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f74a5cdf-7d5f-4992-aceb-be966c15bcd4 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:28.568601 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9775ee6b-0c59-44b6-9ce2-ca308216a56b None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:28.577597 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-f74a5cdf-7d5f-4992-aceb-be966c15bcd4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:31:28.577597 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:28.577597 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:28.577597 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:28.577597 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:28.577597 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:28.577597 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:28.577597 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:28.577597 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:28.577597 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:28.577597 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:28.577597 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:28.577597 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 09:31:28.577597 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 09:31:28.577597 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 09:31:28.578488 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 09:31:28.578488 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:28.578488 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:28.578488 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:28.578488 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:28.578488 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:28.578488 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:28.578488 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:28.578488 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:28.578488 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:28.578488 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:28.578488 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:28.578488 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:28.578488 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:28.578488 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:28.578488 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:28.578488 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:28.579329 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 09:31:28.579329 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 09:31:28.579329 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 09:31:28.579329 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 09:31:28.579329 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 09:31:28.579329 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 09:31:28.579329 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:31:28.579329 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:28.581756 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 271/534] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:31:28 2022] GET /identity/v3/projects/service => generated 88 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:28.582862 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74a5cdf-7d5f-4992-aceb-be966c15bcd4 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:28.610563 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f6032d-3fc6-4437-812f-b51c91f20d02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:28.611456 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f6032d-3fc6-4437-812f-b51c91f20d02 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:28.611746 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f6032d-3fc6-4437-812f-b51c91f20d02 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:28.612020 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f6032d-3fc6-4437-812f-b51c91f20d02 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:28.612861 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90f6032d-3fc6-4437-812f-b51c91f20d02 None admin] RBAC: Adding query filter params (role.id=e5d8d090857a4a018d0ec3c79ece6472, scope.project.id=455d1c826b974bc38497d18a516867c7, user.id=f4b2ea9aa31445d69e0c1d96b5657708) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:28.613176 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90f6032d-3fc6-4437-812f-b51c91f20d02 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:28.614656 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90f6032d-3fc6-4437-812f-b51c91f20d02 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:28.621841 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 264/535] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:31:28 2022] GET /identity/v3/role_assignments?user.id=f4b2ea9aa31445d69e0c1d96b5657708&scope.project.id=455d1c826b974bc38497d18a516867c7&role.id=e5d8d090857a4a018d0ec3c79ece6472 => generated 259 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:28.622813 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f6032d-3fc6-4437-812f-b51c91f20d02 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:28.695862 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:28.749187 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f6032d-3fc6-4437-812f-b51c91f20d02 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:28.760686 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:28.793427 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b30b6f-09ab-461c-9965-e5de36197001 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:28.794368 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b30b6f-09ab-461c-9965-e5de36197001 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:28.795042 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b30b6f-09ab-461c-9965-e5de36197001 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:28.795467 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b30b6f-09ab-461c-9965-e5de36197001 None admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:28.796337 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b30b6f-09ab-461c-9965-e5de36197001 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:28.796718 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74a5cdf-7d5f-4992-aceb-be966c15bcd4 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:28.797055 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b30b6f-09ab-461c-9965-e5de36197001 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:28.798873 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b30b6f-09ab-461c-9965-e5de36197001 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:28.816188 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 265/536] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Oct 28 09:31:28 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:28.823132 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b30b6f-09ab-461c-9965-e5de36197001 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:28.909797 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1610a76-05a5-4e9f-af5e-0a5e64947a86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:28.910718 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1610a76-05a5-4e9f-af5e-0a5e64947a86 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:28.911145 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1610a76-05a5-4e9f-af5e-0a5e64947a86 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:28.911682 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1610a76-05a5-4e9f-af5e-0a5e64947a86 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:28.912475 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1610a76-05a5-4e9f-af5e-0a5e64947a86 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:28.913752 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1610a76-05a5-4e9f-af5e-0a5e64947a86 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:28.915347 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1610a76-05a5-4e9f-af5e-0a5e64947a86 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:28.944306 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 272/537] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 362 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:28.952892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1610a76-05a5-4e9f-af5e-0a5e64947a86 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:28.981525 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b30b6f-09ab-461c-9965-e5de36197001 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:29.072016 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d5eef6-8b52-489f-8194-842ca0a8ca6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:29.072645 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d5eef6-8b52-489f-8194-842ca0a8ca6b None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:29.072929 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d5eef6-8b52-489f-8194-842ca0a8ca6b None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:29.073206 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d5eef6-8b52-489f-8194-842ca0a8ca6b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:29.073940 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9d5eef6-8b52-489f-8194-842ca0a8ca6b None admin] RBAC: Adding query filter params (role.id=e5d8d090857a4a018d0ec3c79ece6472, scope.project.id=455d1c826b974bc38497d18a516867c7, user.id=1021f1cc73b54d6eaa647a5f49815ef8) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:29.074251 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9d5eef6-8b52-489f-8194-842ca0a8ca6b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:29.075955 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9d5eef6-8b52-489f-8194-842ca0a8ca6b None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:29.081442 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1610a76-05a5-4e9f-af5e-0a5e64947a86 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:29.086546 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 266/538] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:31:28 2022] GET /identity/v3/role_assignments?user.id=1021f1cc73b54d6eaa647a5f49815ef8&scope.project.id=455d1c826b974bc38497d18a516867c7&role.id=e5d8d090857a4a018d0ec3c79ece6472 => generated 259 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:29.120053 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:29.125658 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa08969-1522-49f8-8b26-2c29884006d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:29.126319 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa08969-1522-49f8-8b26-2c29884006d3 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:29.127039 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa08969-1522-49f8-8b26-2c29884006d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:29.127323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa08969-1522-49f8-8b26-2c29884006d3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:29.136182 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa08969-1522-49f8-8b26-2c29884006d3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:29.137677 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa08969-1522-49f8-8b26-2c29884006d3 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:29.151124 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-6aa08969-1522-49f8-8b26-2c29884006d3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:31:29.151124 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:29.151124 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:29.151124 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:29.151124 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:29.151124 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:29.151124 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:29.151124 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:29.151124 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:29.151124 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:29.151124 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:29.151124 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:29.151124 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 09:31:29.151124 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 09:31:29.151124 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 09:31:29.152040 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 09:31:29.152040 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:29.152040 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:29.152040 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:29.152040 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:29.152040 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:29.152040 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:29.152040 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:29.152040 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:29.152040 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:29.152040 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:29.152040 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:29.152040 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:29.152040 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:29.152040 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:29.152040 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:29.152040 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:29.153323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 09:31:29.153323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 09:31:29.153323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 09:31:29.153323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 09:31:29.153323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 09:31:29.153323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 09:31:29.153323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:31:29.153323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:29.153323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 273/539] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:31:28 2022] GET /identity/v3/projects/service => generated 88 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:31:29.159039 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d5eef6-8b52-489f-8194-842ca0a8ca6b None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:29.196461 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:29.318781 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d5eef6-8b52-489f-8194-842ca0a8ca6b None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:29.359496 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ef6b9c-9a34-49f1-9f37-0611d92b994e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:29.360176 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ef6b9c-9a34-49f1-9f37-0611d92b994e None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:29.360466 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ef6b9c-9a34-49f1-9f37-0611d92b994e None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:29.360743 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ef6b9c-9a34-49f1-9f37-0611d92b994e None admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:29.361559 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ef6b9c-9a34-49f1-9f37-0611d92b994e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:29.361878 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ef6b9c-9a34-49f1-9f37-0611d92b994e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:29.363372 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ef6b9c-9a34-49f1-9f37-0611d92b994e None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:29.378936 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 267/540] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Oct 28 09:31:29 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:29.385758 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa08969-1522-49f8-8b26-2c29884006d3 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:29.512999 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa08969-1522-49f8-8b26-2c29884006d3 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:29.557993 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c513fc1f-13ea-4519-9456-b6831cca280c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:29.558652 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c513fc1f-13ea-4519-9456-b6831cca280c None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:29.558936 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c513fc1f-13ea-4519-9456-b6831cca280c None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:29.559211 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c513fc1f-13ea-4519-9456-b6831cca280c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:29.559949 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c513fc1f-13ea-4519-9456-b6831cca280c None admin] RBAC: Adding query filter params (role.id=e5d8d090857a4a018d0ec3c79ece6472, scope.project.id=455d1c826b974bc38497d18a516867c7, user.id=1db6b98e87e64584932447c82fefc3b7) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:29.560259 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c513fc1f-13ea-4519-9456-b6831cca280c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:29.561690 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c513fc1f-13ea-4519-9456-b6831cca280c None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:29.573311 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 274/541] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:31:29 2022] GET /identity/v3/role_assignments?user.id=1db6b98e87e64584932447c82fefc3b7&scope.project.id=455d1c826b974bc38497d18a516867c7&role.id=e5d8d090857a4a018d0ec3c79ece6472 => generated 259 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:30.208947 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5e40d8-4c37-4e0b-8b35-ba82f4a565a5 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:30.209464 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5e40d8-4c37-4e0b-8b35-ba82f4a565a5 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:30.209571 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5e40d8-4c37-4e0b-8b35-ba82f4a565a5 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:30.211662 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 268/542] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:30.217212 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1cbe9a-9ac2-4b46-996d-7d1a4063d835 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:30.217505 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1cbe9a-9ac2-4b46-996d-7d1a4063d835 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:30.217789 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1cbe9a-9ac2-4b46-996d-7d1a4063d835 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:30.236471 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-ba1cbe9a-9ac2-4b46-996d-7d1a4063d835 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:30.237681 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ba1cbe9a-9ac2-4b46-996d-7d1a4063d835 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:30.292788 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b0db3a-a667-493a-bf2f-9f092ea1c345 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:30.293169 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b0db3a-a667-493a-bf2f-9f092ea1c345 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:30.293435 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b0db3a-a667-493a-bf2f-9f092ea1c345 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:30.294718 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 269/543] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 09:31:30.295735 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:30.304681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2c6b07-fcac-4077-83b7-f216bdd27a31 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:30.304966 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2c6b07-fcac-4077-83b7-f216bdd27a31 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:30.305248 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2c6b07-fcac-4077-83b7-f216bdd27a31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:30.329899 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-1a2c6b07-fcac-4077-83b7-f216bdd27a31 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:30.331599 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-1a2c6b07-fcac-4077-83b7-f216bdd27a31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:30.344346 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 275/544] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 09:31:30.425613 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 270/545] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:31:30.699437 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1cbe9a-9ac2-4b46-996d-7d1a4063d835 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:30.757499 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a2c6b07-fcac-4077-83b7-f216bdd27a31 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:30.760946 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:30.828423 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1cbe9a-9ac2-4b46-996d-7d1a4063d835 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:30.864872 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21eec6cd-1d00-4540-89d6-eca2eab8ee46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:30.865906 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21eec6cd-1d00-4540-89d6-eca2eab8ee46 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:30.866958 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21eec6cd-1d00-4540-89d6-eca2eab8ee46 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:30.867469 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21eec6cd-1d00-4540-89d6-eca2eab8ee46 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:30.889486 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21eec6cd-1d00-4540-89d6-eca2eab8ee46 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:31:30.896000 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-21eec6cd-1d00-4540-89d6-eca2eab8ee46 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 09:31:30.896000 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:30.896000 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:30.896000 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:30.896000 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:30.896000 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:30.896000 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:30.896000 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:30.896000 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:30.896000 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:30.896000 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:30.896000 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:30.896000 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:31:30.896000 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:31:30.896000 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:31:30.896000 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:31:30.896892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:31:30.896892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:31:30.896892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:31:30.896892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:31:30.896892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:30.896892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:30.896892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:30.896892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:30.896892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:30.896892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:30.896892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:30.896892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:30.896892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:30.896892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:30.896892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:30.896892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:30.898200 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:30.898200 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:30.898200 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:30.898200 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:30.898200 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:31:30.898200 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:31:30.898200 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:31:30.898200 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:31:30.898200 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:31:30.898200 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:31:30.898200 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 09:31:30.898200 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:30.907969 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 276/546] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Oct 28 09:31:30 2022] GET /identity/v3/roles/service => generated 85 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:30.913973 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21eec6cd-1d00-4540-89d6-eca2eab8ee46 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:30.943705 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a2c6b07-fcac-4077-83b7-f216bdd27a31 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:30.995743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:31.016680 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-881b7c4d-1914-4188-8a59-5675f429759c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:31.017327 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881b7c4d-1914-4188-8a59-5675f429759c None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:31.017608 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881b7c4d-1914-4188-8a59-5675f429759c None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:31.017883 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-881b7c4d-1914-4188-8a59-5675f429759c None admin] PATH_INFO: `/v3/roles/service` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:31.039379 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-881b7c4d-1914-4188-8a59-5675f429759c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:31:31.049293 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-881b7c4d-1914-4188-8a59-5675f429759c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 09:31:31.049293 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:31.049293 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:31.049293 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:31.049293 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:31.049293 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:31.049293 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:31.049293 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:31.049293 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:31.049293 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:31.049293 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:31.049293 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:31.049293 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:31:31.049293 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:31:31.049293 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:31:31.049293 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:31:31.050235 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:31:31.050235 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:31:31.050235 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:31:31.050235 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:31:31.050235 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:31.050235 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:31.050235 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:31.050235 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:31.050235 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:31.050235 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:31.050235 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:31.050235 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:31.050235 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:31.050235 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:31.050235 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:31.050235 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:31.051053 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:31.051053 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:31.051053 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:31.051053 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:31.051053 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:31:31.051053 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:31:31.051053 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:31:31.051053 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:31:31.051053 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:31:31.051053 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:31:31.051053 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 09:31:31.051053 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:31.052606 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 271/547] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Oct 28 09:31:30 2022] GET /identity/v3/roles/service => generated 85 bytes in 296 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:31:31.060421 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-881b7c4d-1914-4188-8a59-5675f429759c None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:31.088719 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21eec6cd-1d00-4540-89d6-eca2eab8ee46 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:31.128491 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c43d108e-4011-488f-88ef-1cdde5366695 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:31.129548 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43d108e-4011-488f-88ef-1cdde5366695 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:31.130114 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43d108e-4011-488f-88ef-1cdde5366695 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:31.130620 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43d108e-4011-488f-88ef-1cdde5366695 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:31.131481 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c43d108e-4011-488f-88ef-1cdde5366695 None admin] RBAC: Adding query filter params (name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:31.132081 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c43d108e-4011-488f-88ef-1cdde5366695 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:31.133642 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c43d108e-4011-488f-88ef-1cdde5366695 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:31.143266 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 277/548] 10.222.0.25 () {56 vars in 1103 bytes} [Fri Oct 28 09:31:30 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:31:31.149226 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c43d108e-4011-488f-88ef-1cdde5366695 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:31.220162 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:31.260358 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c43d108e-4011-488f-88ef-1cdde5366695 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:31.270829 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:31.301089 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-881b7c4d-1914-4188-8a59-5675f429759c None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:31.305428 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec5db73-9767-4102-b1b8-18e45c74c791 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:31.306421 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec5db73-9767-4102-b1b8-18e45c74c791 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:31.306905 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec5db73-9767-4102-b1b8-18e45c74c791 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:31.307462 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec5db73-9767-4102-b1b8-18e45c74c791 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:31.332456 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ec5db73-9767-4102-b1b8-18e45c74c791 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:31.336031 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ec5db73-9767-4102-b1b8-18e45c74c791 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:31.364195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-3ec5db73-9767-4102-b1b8-18e45c74c791 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:31.364195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:31.364195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:31.364195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:31.364195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:31.364195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:31.364195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:31.364195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:31.364195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:31.364195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:31.364195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:31.364195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:31.364195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:31.364195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:31.364195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:31.364195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:31.365282 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:31.365282 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:31.365282 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:31.365282 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:31.365282 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:31.365282 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:31.365282 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:31.365282 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:31.365282 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:31.365282 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:31.365282 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:31.365282 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:31.365282 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:31.365282 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:31.365282 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:31.365282 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:31.365282 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:31.366115 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:31.366115 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:31.366115 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:31.371073 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 278/549] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:31:31.377137 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec5db73-9767-4102-b1b8-18e45c74c791 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:31.387183 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540c4cfd-3e13-40d9-9c10-409ee49f524c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:31.387815 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540c4cfd-3e13-40d9-9c10-409ee49f524c None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:31.388094 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540c4cfd-3e13-40d9-9c10-409ee49f524c None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:31.388371 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540c4cfd-3e13-40d9-9c10-409ee49f524c None admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:31.389053 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-540c4cfd-3e13-40d9-9c10-409ee49f524c None admin] RBAC: Adding query filter params (name=service) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:31.389358 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-540c4cfd-3e13-40d9-9c10-409ee49f524c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:31.390782 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-540c4cfd-3e13-40d9-9c10-409ee49f524c None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:31.405081 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 272/550] 10.222.0.25 () {56 vars in 1103 bytes} [Fri Oct 28 09:31:31 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 345 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:31:31.423531 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540c4cfd-3e13-40d9-9c10-409ee49f524c None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:31.504331 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ec5db73-9767-4102-b1b8-18e45c74c791 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:31.544864 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895af06f-5b8a-4568-8c59-de42db4b63cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:31.545854 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895af06f-5b8a-4568-8c59-de42db4b63cc None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:31.546337 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895af06f-5b8a-4568-8c59-de42db4b63cc None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:31.546864 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895af06f-5b8a-4568-8c59-de42db4b63cc None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:31.548041 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-895af06f-5b8a-4568-8c59-de42db4b63cc None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:31.548547 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-895af06f-5b8a-4568-8c59-de42db4b63cc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:31.550117 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-895af06f-5b8a-4568-8c59-de42db4b63cc None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:31.579097 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 279/551] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:31.580681 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4610177-21e4-4cc1-94e0-b19ebb4cf34e None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:31.581971 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4610177-21e4-4cc1-94e0-b19ebb4cf34e None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:31.582272 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4610177-21e4-4cc1-94e0-b19ebb4cf34e None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:31.583886 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 280/552] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:31.584542 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4610177-21e4-4cc1-94e0-b19ebb4cf34e None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:31.613299 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:31.636993 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540c4cfd-3e13-40d9-9c10-409ee49f524c None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:31.665741 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:31.681599 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b7b91d3-8408-4f68-ad87-8b449c6ae4e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:31.682118 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7b91d3-8408-4f68-ad87-8b449c6ae4e8 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:31.682418 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7b91d3-8408-4f68-ad87-8b449c6ae4e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:31.682719 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7b91d3-8408-4f68-ad87-8b449c6ae4e8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:31.694578 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b7b91d3-8408-4f68-ad87-8b449c6ae4e8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:31.696071 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b7b91d3-8408-4f68-ad87-8b449c6ae4e8 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:31.711361 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-1b7b91d3-8408-4f68-ad87-8b449c6ae4e8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:31.711361 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:31.711361 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:31.711361 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:31.711361 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:31.711361 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:31.711361 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:31.711361 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:31.711361 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:31.711361 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:31.711361 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:31.711361 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:31.711361 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:31.711361 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:31.711361 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:31.711361 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:31.712597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:31.712597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:31.712597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:31.712597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:31.712597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:31.712597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:31.712597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:31.712597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:31.712597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:31.712597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:31.712597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:31.712597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:31.712597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:31.712597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:31.712597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:31.712597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:31.712597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:31.714046 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:31.714046 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:31.714046 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:31.714046 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 273/553] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 290 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:31.714046 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df162667-1444-4290-bad5-cca5add42866 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:31.714319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df162667-1444-4290-bad5-cca5add42866 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:31.714417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df162667-1444-4290-bad5-cca5add42866 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:31.728062 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4610177-21e4-4cc1-94e0-b19ebb4cf34e None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:31.735291 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-df162667-1444-4290-bad5-cca5add42866 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:31.736983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-df162667-1444-4290-bad5-cca5add42866 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:31.771379 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b901180-ed62-4d59-8fb4-8ca2081d4e84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:31.772008 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b901180-ed62-4d59-8fb4-8ca2081d4e84 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:31.772284 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b901180-ed62-4d59-8fb4-8ca2081d4e84 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:31.773081 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b901180-ed62-4d59-8fb4-8ca2081d4e84 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:31.780542 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b901180-ed62-4d59-8fb4-8ca2081d4e84 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:31.781970 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b901180-ed62-4d59-8fb4-8ca2081d4e84 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:31.796013 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-7b901180-ed62-4d59-8fb4-8ca2081d4e84 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 28 09:31:31.796013 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:31.796013 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:31.796013 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:31.796013 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:31.796013 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:31.796013 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:31.796013 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:31.796013 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:31.796013 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:31.796013 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:31.796013 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:31.796013 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 09:31:31.796013 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 09:31:31.796013 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 09:31:31.796013 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 09:31:31.797181 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:31.797181 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:31.797181 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 09:31:31.797181 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:31:31.797181 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 09:31:31.797181 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:31:31.797181 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:31.797181 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:31.797181 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:31.797181 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:31.797181 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:31.797181 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:31.797181 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:31.797181 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:31.797181 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:31.797181 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:31.797181 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:31.797181 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:31.798736 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:31.798736 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:31.798736 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 09:31:31.798736 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 09:31:31.798736 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 09:31:31.798736 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 09:31:31.798736 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 09:31:31.798736 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 09:31:31.798736 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 28 09:31:31.798736 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:31.798736 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 281/554] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Oct 28 09:31:31 2022] GET /identity/v3/users/swift => generated 83 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:31.798736 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d33a51-ba6e-4c4b-aa66-d2b772849255 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:31.798736 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d33a51-ba6e-4c4b-aa66-d2b772849255 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:31.799444 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d33a51-ba6e-4c4b-aa66-d2b772849255 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:31.802531 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 282/555] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:31 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:31.802531 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d33a51-ba6e-4c4b-aa66-d2b772849255 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:31.816747 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 274/556] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:31:31.827436 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df162667-1444-4290-bad5-cca5add42866 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:31.913373 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:31.940403 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d33a51-ba6e-4c4b-aa66-d2b772849255 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:31.941939 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df162667-1444-4290-bad5-cca5add42866 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:31.950460 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:31.981920 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee536f4-50de-4e0b-b26f-e14f0e3374db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:31.982749 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee536f4-50de-4e0b-b26f-e14f0e3374db None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:31.983127 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee536f4-50de-4e0b-b26f-e14f0e3374db None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:31.983489 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee536f4-50de-4e0b-b26f-e14f0e3374db None admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:31.984352 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee536f4-50de-4e0b-b26f-e14f0e3374db None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:31.984901 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee536f4-50de-4e0b-b26f-e14f0e3374db None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:31.986159 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467ff23a-5613-4e7f-b2d3-bf4dce4075a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:31.986615 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fee536f4-50de-4e0b-b26f-e14f0e3374db None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:31.987187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467ff23a-5613-4e7f-b2d3-bf4dce4075a4 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:31.987626 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467ff23a-5613-4e7f-b2d3-bf4dce4075a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:31.988057 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467ff23a-5613-4e7f-b2d3-bf4dce4075a4 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:31.988978 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-467ff23a-5613-4e7f-b2d3-bf4dce4075a4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:31.989453 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-467ff23a-5613-4e7f-b2d3-bf4dce4075a4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:31.991233 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-467ff23a-5613-4e7f-b2d3-bf4dce4075a4 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:32.009691 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 283/557] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:32.011530 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fea8770-65fd-4e3f-9a81-ce246d2cd05b None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:32.012001 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fea8770-65fd-4e3f-9a81-ce246d2cd05b None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:32.012538 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fea8770-65fd-4e3f-9a81-ce246d2cd05b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:32.014005 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 275/558] 10.222.0.25 () {56 vars in 1135 bytes} [Fri Oct 28 09:31:31 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:32.015065 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee536f4-50de-4e0b-b26f-e14f0e3374db None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:32.035157 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-7fea8770-65fd-4e3f-9a81-ce246d2cd05b None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:32.036969 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7fea8770-65fd-4e3f-9a81-ce246d2cd05b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:32.124506 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 284/559] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:31:32.125375 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fea8770-65fd-4e3f-9a81-ce246d2cd05b None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:32.141225 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fee536f4-50de-4e0b-b26f-e14f0e3374db None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:32.183671 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473bfab8-6cd2-445d-b949-f477e1b56c1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:32.184236 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473bfab8-6cd2-445d-b949-f477e1b56c1e None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:32.184516 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473bfab8-6cd2-445d-b949-f477e1b56c1e None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:32.184792 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473bfab8-6cd2-445d-b949-f477e1b56c1e None admin] PATH_INFO: `/v3/users/nova` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:32.193519 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473bfab8-6cd2-445d-b949-f477e1b56c1e None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:32.195241 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473bfab8-6cd2-445d-b949-f477e1b56c1e None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:32.208603 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-473bfab8-6cd2-445d-b949-f477e1b56c1e None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 28 09:31:32.208603 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:32.208603 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:32.208603 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:32.208603 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:32.208603 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:32.208603 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:32.208603 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:32.208603 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:32.208603 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:32.208603 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:32.208603 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:32.208603 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 09:31:32.208603 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 09:31:32.208603 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 09:31:32.208603 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 09:31:32.209660 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:32.209660 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:32.209660 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 09:31:32.209660 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:31:32.209660 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 09:31:32.209660 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:31:32.209660 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:32.209660 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:32.209660 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:32.209660 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:32.209660 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:32.209660 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:32.209660 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:32.209660 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:32.209660 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:32.209660 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:32.209660 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:32.209660 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:32.211308 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:32.211308 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:32.211308 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 09:31:32.211308 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 09:31:32.211308 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 09:31:32.211308 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 09:31:32.211308 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 09:31:32.211308 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 09:31:32.211308 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 28 09:31:32.211308 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:32.211308 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 276/560] 10.222.0.25 () {56 vars in 1093 bytes} [Fri Oct 28 09:31:32 2022] GET /identity/v3/users/nova => generated 82 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:32.211308 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473bfab8-6cd2-445d-b949-f477e1b56c1e None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:32.232031 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:32.243787 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:32.281060 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fea8770-65fd-4e3f-9a81-ce246d2cd05b None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:32.336243 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da024bb-797c-4b92-99c9-c56d493eac36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:32.337479 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da024bb-797c-4b92-99c9-c56d493eac36 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:32.337928 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da024bb-797c-4b92-99c9-c56d493eac36 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:32.338520 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da024bb-797c-4b92-99c9-c56d493eac36 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:32.339083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473bfab8-6cd2-445d-b949-f477e1b56c1e None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:32.379035 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-859a8e1b-e854-4a73-ba8d-80f15e6a9aac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:32.379802 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859a8e1b-e854-4a73-ba8d-80f15e6a9aac None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:32.380148 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859a8e1b-e854-4a73-ba8d-80f15e6a9aac None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:32.380426 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859a8e1b-e854-4a73-ba8d-80f15e6a9aac None admin] PATH_INFO: `/v3/roles/service` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:32.393624 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-859a8e1b-e854-4a73-ba8d-80f15e6a9aac None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:31:32.399865 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-859a8e1b-e854-4a73-ba8d-80f15e6a9aac None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 09:31:32.399865 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:32.399865 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:32.399865 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:32.399865 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:32.399865 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:32.399865 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:32.399865 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:32.399865 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:32.399865 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:32.399865 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:32.399865 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:32.399865 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:31:32.399865 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:31:32.399865 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:31:32.399865 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:31:32.400901 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:31:32.400901 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:31:32.400901 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:31:32.400901 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:31:32.400901 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:32.400901 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:32.400901 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:32.400901 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:32.400901 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:32.400901 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:32.400901 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:32.400901 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:32.400901 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:32.400901 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:32.400901 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:32.400901 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:32.401758 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:32.401758 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:32.401758 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:32.401758 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:32.401758 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:31:32.401758 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:31:32.401758 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:31:32.401758 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:31:32.401758 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:31:32.401758 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:31:32.401758 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 09:31:32.401758 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:32.401758 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 277/561] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Oct 28 09:31:32 2022] GET /identity/v3/roles/service => generated 85 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:32.403174 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-859a8e1b-e854-4a73-ba8d-80f15e6a9aac None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:32.433298 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da024bb-797c-4b92-99c9-c56d493eac36 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:32.434895 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da024bb-797c-4b92-99c9-c56d493eac36 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:32.498540 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:32.508794 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:32.515995 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-859a8e1b-e854-4a73-ba8d-80f15e6a9aac None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:32.542279 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 285/562] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:31:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 417 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 09:31:32.543582 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a99daf-8a20-43e9-9e86-eb4179910cd8 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:32.543915 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a99daf-8a20-43e9-9e86-eb4179910cd8 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:32.544149 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a99daf-8a20-43e9-9e86-eb4179910cd8 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:32.545392 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 286/563] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:32.546088 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a99daf-8a20-43e9-9e86-eb4179910cd8 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:32.558197 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eff5ad1-40e3-4185-8ca7-2b9f87e18ba4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:32.559210 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eff5ad1-40e3-4185-8ca7-2b9f87e18ba4 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:32.559685 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eff5ad1-40e3-4185-8ca7-2b9f87e18ba4 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:32.560160 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eff5ad1-40e3-4185-8ca7-2b9f87e18ba4 None admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:32.561492 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eff5ad1-40e3-4185-8ca7-2b9f87e18ba4 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:32.562012 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eff5ad1-40e3-4185-8ca7-2b9f87e18ba4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:32.563695 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eff5ad1-40e3-4185-8ca7-2b9f87e18ba4 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:32.586974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 278/564] 10.222.0.25 () {56 vars in 1133 bytes} [Fri Oct 28 09:31:32 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:32.588100 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eff5ad1-40e3-4185-8ca7-2b9f87e18ba4 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:32.661580 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a99daf-8a20-43e9-9e86-eb4179910cd8 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:32.696673 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8362767c-6d89-4190-9759-52041c6dc5dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:32.697202 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8362767c-6d89-4190-9759-52041c6dc5dc None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:32.697481 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8362767c-6d89-4190-9759-52041c6dc5dc None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:32.697756 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8362767c-6d89-4190-9759-52041c6dc5dc None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:32.698439 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8362767c-6d89-4190-9759-52041c6dc5dc None admin] RBAC: Adding query filter params (name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:32.698749 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8362767c-6d89-4190-9759-52041c6dc5dc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:32.699023 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eff5ad1-40e3-4185-8ca7-2b9f87e18ba4 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:32.700106 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8362767c-6d89-4190-9759-52041c6dc5dc None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:32.709618 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 287/565] 10.222.0.25 () {56 vars in 1103 bytes} [Fri Oct 28 09:31:32 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:32.710628 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8362767c-6d89-4190-9759-52041c6dc5dc None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:32.735674 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3415aab8-265b-48e8-a4fb-78728781515d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:32.736308 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3415aab8-265b-48e8-a4fb-78728781515d None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:32.736627 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3415aab8-265b-48e8-a4fb-78728781515d None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:32.736914 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3415aab8-265b-48e8-a4fb-78728781515d None admin] PATH_INFO: `/v3/roles/service` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:32.743380 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:32.747978 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3415aab8-265b-48e8-a4fb-78728781515d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:31:32.752948 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-3415aab8-265b-48e8-a4fb-78728781515d None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 09:31:32.752948 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:32.752948 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:32.752948 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:32.752948 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:32.752948 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:32.752948 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:32.752948 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:32.752948 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:32.752948 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:32.752948 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:32.752948 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:32.752948 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:31:32.752948 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:31:32.752948 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:31:32.752948 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:31:32.754081 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:31:32.754081 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:31:32.754081 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:31:32.754081 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:31:32.754081 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:32.754081 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:32.754081 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:32.754081 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:32.754081 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:32.754081 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:32.754081 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:32.754081 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:32.754081 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:32.754081 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:32.754081 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:32.754081 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:32.754962 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:32.754962 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:32.754962 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:32.754962 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:32.754962 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:31:32.754962 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:31:32.754962 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:31:32.754962 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:31:32.754962 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:31:32.754962 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:31:32.754962 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 09:31:32.754962 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:32.754962 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 279/566] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Oct 28 09:31:32 2022] GET /identity/v3/roles/service => generated 85 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:32.756145 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84e20a8-084b-45df-bc34-a2e65c0db0f8 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:32.756430 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84e20a8-084b-45df-bc34-a2e65c0db0f8 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:32.756708 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84e20a8-084b-45df-bc34-a2e65c0db0f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:32.763521 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:32.775211 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-d84e20a8-084b-45df-bc34-a2e65c0db0f8 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:32.776427 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-d84e20a8-084b-45df-bc34-a2e65c0db0f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:32.818972 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8362767c-6d89-4190-9759-52041c6dc5dc None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:32.845713 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 280/567] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:31:32.846609 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84e20a8-084b-45df-bc34-a2e65c0db0f8 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:32.866321 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339d4b60-fd0d-454e-9317-3433517ed409 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:32.867513 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339d4b60-fd0d-454e-9317-3433517ed409 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:32.867856 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339d4b60-fd0d-454e-9317-3433517ed409 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:32.868135 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339d4b60-fd0d-454e-9317-3433517ed409 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:32.879287 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-339d4b60-fd0d-454e-9317-3433517ed409 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:32.880756 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-339d4b60-fd0d-454e-9317-3433517ed409 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:32.896104 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-339d4b60-fd0d-454e-9317-3433517ed409 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:32.896104 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:32.896104 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:32.896104 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:32.896104 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:32.896104 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:32.896104 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:32.896104 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:32.896104 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:32.896104 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:32.896104 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:32.896104 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:32.896104 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:32.896104 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:32.896104 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:32.896104 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:32.897123 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:32.897123 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:32.897123 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:32.897123 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:32.897123 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:32.897123 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:32.897123 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:32.897123 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:32.897123 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:32.897123 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:32.897123 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:32.897123 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:32.897123 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:32.897123 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:32.897123 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:32.897123 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:32.897123 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:32.898058 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:32.898058 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:32.898058 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:32.898058 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 288/568] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:32.898058 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339d4b60-fd0d-454e-9317-3433517ed409 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:32.958868 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84e20a8-084b-45df-bc34-a2e65c0db0f8 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:32.985030 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:32.991949 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:32.997634 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b0f1e9-2abe-4763-9982-5e5bbbf54061 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:32.998280 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b0f1e9-2abe-4763-9982-5e5bbbf54061 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:32.998592 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b0f1e9-2abe-4763-9982-5e5bbbf54061 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:32.998869 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b0f1e9-2abe-4763-9982-5e5bbbf54061 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:33.002701 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339d4b60-fd0d-454e-9317-3433517ed409 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:33.010617 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60b0f1e9-2abe-4763-9982-5e5bbbf54061 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:33.012129 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60b0f1e9-2abe-4763-9982-5e5bbbf54061 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:33.029211 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-60b0f1e9-2abe-4763-9982-5e5bbbf54061 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:33.029211 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:33.029211 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:33.029211 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:33.029211 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:33.029211 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:33.029211 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:33.029211 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:33.029211 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:33.029211 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:33.029211 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:33.029211 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:33.029211 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:33.029211 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:33.029211 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:33.029211 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:33.030261 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:33.030261 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:33.030261 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:33.030261 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:33.030261 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:33.030261 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:33.030261 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:33.030261 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:33.030261 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:33.030261 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:33.030261 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:33.030261 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:33.030261 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:33.030261 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:33.030261 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:33.030261 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:33.030261 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:33.031186 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:33.031186 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:33.031186 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:33.031186 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 281/569] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:33.031433 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b0f1e9-2abe-4763-9982-5e5bbbf54061 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:33.046224 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bee06c3-af9a-40a7-8761-cc7764cca790 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:33.046981 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bee06c3-af9a-40a7-8761-cc7764cca790 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:33.047259 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bee06c3-af9a-40a7-8761-cc7764cca790 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:33.047532 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bee06c3-af9a-40a7-8761-cc7764cca790 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:33.059218 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bee06c3-af9a-40a7-8761-cc7764cca790 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:33.060686 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bee06c3-af9a-40a7-8761-cc7764cca790 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:33.078671 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-4bee06c3-af9a-40a7-8761-cc7764cca790 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:33.078671 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:33.078671 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:33.078671 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:33.078671 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:33.078671 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:33.078671 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:33.078671 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:33.078671 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:33.078671 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:33.078671 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:33.078671 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:33.078671 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:33.078671 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:33.078671 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:33.078671 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:33.079737 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:33.079737 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:33.079737 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:33.079737 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:33.079737 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:33.079737 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:33.079737 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:33.079737 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:33.079737 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:33.079737 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:33.079737 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:33.079737 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:33.079737 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:33.079737 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:33.079737 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:33.079737 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:33.079737 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:33.080673 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:33.080673 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:33.080673 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:33.084637 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 289/570] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:33.085732 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bee06c3-af9a-40a7-8761-cc7764cca790 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:33.145413 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b0f1e9-2abe-4763-9982-5e5bbbf54061 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:33.181216 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c569bbb-351e-4ce5-9bca-c69a55d820cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:33.181832 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c569bbb-351e-4ce5-9bca-c69a55d820cf None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:33.182112 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c569bbb-351e-4ce5-9bca-c69a55d820cf None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:33.182413 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c569bbb-351e-4ce5-9bca-c69a55d820cf None admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:33.183084 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c569bbb-351e-4ce5-9bca-c69a55d820cf None admin] RBAC: Adding query filter params (name=service) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:33.183387 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c569bbb-351e-4ce5-9bca-c69a55d820cf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:33.184786 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c569bbb-351e-4ce5-9bca-c69a55d820cf None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:33.193292 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 282/571] 10.222.0.25 () {56 vars in 1103 bytes} [Fri Oct 28 09:31:33 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:33.193971 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c569bbb-351e-4ce5-9bca-c69a55d820cf None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:33.201169 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bee06c3-af9a-40a7-8761-cc7764cca790 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:33.237241 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68865364-5fdf-496e-8155-5b4631c9a1fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:33.237776 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68865364-5fdf-496e-8155-5b4631c9a1fa None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:33.238051 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68865364-5fdf-496e-8155-5b4631c9a1fa None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:33.238318 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68865364-5fdf-496e-8155-5b4631c9a1fa None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:33.239012 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68865364-5fdf-496e-8155-5b4631c9a1fa None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:33.239314 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68865364-5fdf-496e-8155-5b4631c9a1fa None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:33.240689 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68865364-5fdf-496e-8155-5b4631c9a1fa None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:33.244041 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:33.244619 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:33.254498 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 290/572] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:33.255374 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68865364-5fdf-496e-8155-5b4631c9a1fa None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:33.302773 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c569bbb-351e-4ce5-9bca-c69a55d820cf None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:33.342169 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10db49a1-4715-4da1-9f7d-b5f334d424a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:33.342811 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10db49a1-4715-4da1-9f7d-b5f334d424a4 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:33.343089 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10db49a1-4715-4da1-9f7d-b5f334d424a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:33.343361 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10db49a1-4715-4da1-9f7d-b5f334d424a4 None admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:33.344000 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10db49a1-4715-4da1-9f7d-b5f334d424a4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:33.344298 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10db49a1-4715-4da1-9f7d-b5f334d424a4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:33.345659 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10db49a1-4715-4da1-9f7d-b5f334d424a4 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:33.357912 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 283/573] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:33.358795 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10db49a1-4715-4da1-9f7d-b5f334d424a4 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:33.363899 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68865364-5fdf-496e-8155-5b4631c9a1fa None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:33.401274 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2635d6fd-d675-4207-a389-ac0ee10c3188 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:33.401918 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2635d6fd-d675-4207-a389-ac0ee10c3188 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:33.402197 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2635d6fd-d675-4207-a389-ac0ee10c3188 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:33.402506 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2635d6fd-d675-4207-a389-ac0ee10c3188 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:33.403164 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2635d6fd-d675-4207-a389-ac0ee10c3188 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:33.403465 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2635d6fd-d675-4207-a389-ac0ee10c3188 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:33.404868 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2635d6fd-d675-4207-a389-ac0ee10c3188 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:33.417530 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 291/574] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:33.418378 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2635d6fd-d675-4207-a389-ac0ee10c3188 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:33.468442 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10db49a1-4715-4da1-9f7d-b5f334d424a4 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:33.474706 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:33.476902 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:33.503767 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8decb3aa-609e-4847-b642-1aeb9454bbf1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:33.504317 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8decb3aa-609e-4847-b642-1aeb9454bbf1 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:33.504606 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8decb3aa-609e-4847-b642-1aeb9454bbf1 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:33.504880 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8decb3aa-609e-4847-b642-1aeb9454bbf1 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:33.515114 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8decb3aa-609e-4847-b642-1aeb9454bbf1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:31:33.519807 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2635d6fd-d675-4207-a389-ac0ee10c3188 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:33.519938 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-8decb3aa-609e-4847-b642-1aeb9454bbf1 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 09:31:33.519938 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:33.519938 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:33.519938 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:33.519938 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:33.519938 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:33.519938 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:33.519938 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:33.519938 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:33.519938 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:33.519938 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:33.519938 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:33.519938 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:31:33.519938 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:31:33.519938 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:31:33.519938 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:31:33.520988 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:31:33.520988 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:31:33.520988 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:31:33.520988 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:31:33.520988 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:33.520988 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:33.520988 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:33.520988 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:33.520988 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:33.520988 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:33.520988 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:33.520988 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:33.520988 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:33.520988 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:33.520988 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:33.520988 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:33.522357 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:33.522357 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:33.522357 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:33.522357 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:33.522357 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:31:33.522357 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:31:33.522357 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:31:33.522357 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:31:33.522357 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:31:33.522357 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:31:33.522357 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 09:31:33.522357 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:33.522357 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 284/575] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Oct 28 09:31:33 2022] GET /identity/v3/roles/service => generated 85 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:33.522357 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8decb3aa-609e-4847-b642-1aeb9454bbf1 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:33.555890 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9fde118-bb2d-4f2e-8fb0-1296e8935478 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:33.556520 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9fde118-bb2d-4f2e-8fb0-1296e8935478 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:33.556803 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9fde118-bb2d-4f2e-8fb0-1296e8935478 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:33.557078 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9fde118-bb2d-4f2e-8fb0-1296e8935478 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:33.567669 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9fde118-bb2d-4f2e-8fb0-1296e8935478 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:33.569102 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9fde118-bb2d-4f2e-8fb0-1296e8935478 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:33.586679 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-a9fde118-bb2d-4f2e-8fb0-1296e8935478 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:33.586679 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:33.586679 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:33.586679 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:33.586679 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:33.586679 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:33.586679 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:33.586679 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:33.586679 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:33.586679 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:33.586679 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:33.586679 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:33.586679 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:33.586679 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:33.586679 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:33.586679 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:33.587743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:33.587743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:33.587743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:33.587743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:33.587743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:33.587743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:33.587743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:33.587743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:33.587743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:33.587743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:33.587743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:33.587743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:33.587743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:33.587743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:33.587743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:33.587743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:33.587743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:33.588650 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:33.588650 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:33.588650 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:33.588650 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 292/576] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:33.588650 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9fde118-bb2d-4f2e-8fb0-1296e8935478 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:33.634319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8decb3aa-609e-4847-b642-1aeb9454bbf1 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:33.678217 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d0a6c5-26bb-4fde-a14b-de51248de431 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:33.678883 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d0a6c5-26bb-4fde-a14b-de51248de431 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:33.679156 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d0a6c5-26bb-4fde-a14b-de51248de431 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:33.679428 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d0a6c5-26bb-4fde-a14b-de51248de431 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:33.686011 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d0a6c5-26bb-4fde-a14b-de51248de431 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:33.687503 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54d0a6c5-26bb-4fde-a14b-de51248de431 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:33.693121 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9fde118-bb2d-4f2e-8fb0-1296e8935478 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:33.698992 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-54d0a6c5-26bb-4fde-a14b-de51248de431 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:31:33.698992 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:33.698992 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:33.698992 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:33.698992 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:33.698992 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:33.698992 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:33.698992 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:33.698992 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:33.698992 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:33.698992 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:33.698992 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:33.698992 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 09:31:33.698992 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 09:31:33.698992 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 09:31:33.700073 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 09:31:33.700073 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:33.700073 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:33.700073 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:33.700073 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:33.700073 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:33.700073 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:33.700073 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:33.700073 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:33.700073 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:33.700073 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:33.700073 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:33.700073 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:33.700073 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:33.700073 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:33.700073 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:33.700073 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:33.700958 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 09:31:33.700958 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 09:31:33.700958 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 09:31:33.700958 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 09:31:33.700958 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 09:31:33.700958 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 09:31:33.700958 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:31:33.700958 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:33.700958 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 285/577] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:31:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:33.701461 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d0a6c5-26bb-4fde-a14b-de51248de431 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:33.704299 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:33.729702 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:33.730995 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da91088a-776e-4c53-b924-d3c4ccf741fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:33.731932 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da91088a-776e-4c53-b924-d3c4ccf741fe None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:33.732223 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da91088a-776e-4c53-b924-d3c4ccf741fe None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:33.732501 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da91088a-776e-4c53-b924-d3c4ccf741fe None admin] PATH_INFO: `/v3/projects/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:33.739283 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da91088a-776e-4c53-b924-d3c4ccf741fe None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:33.740708 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da91088a-776e-4c53-b924-d3c4ccf741fe None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:33.752160 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-da91088a-776e-4c53-b924-d3c4ccf741fe None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:31:33.752160 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:33.752160 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:33.752160 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:33.752160 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:33.752160 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:33.752160 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:33.752160 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:33.752160 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:33.752160 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:33.752160 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:33.752160 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:33.752160 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 09:31:33.752160 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 09:31:33.752160 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 09:31:33.753530 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 09:31:33.753530 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:33.753530 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:33.753530 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:33.753530 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:33.753530 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:33.753530 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:33.753530 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:33.753530 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:33.753530 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:33.753530 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:33.753530 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:33.753530 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:33.753530 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:33.753530 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:33.753530 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:33.753530 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:33.754463 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 09:31:33.754463 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 09:31:33.754463 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 09:31:33.754463 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 09:31:33.754463 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 09:31:33.754463 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 09:31:33.754463 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:31:33.754463 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:33.754463 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 293/578] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:31:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:33.754463 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da91088a-776e-4c53-b924-d3c4ccf741fe None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:33.808008 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d0a6c5-26bb-4fde-a14b-de51248de431 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:33.842650 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c15a5b0-5781-409d-8a4c-21f0931c103d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:33.843285 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c15a5b0-5781-409d-8a4c-21f0931c103d None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:33.843564 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c15a5b0-5781-409d-8a4c-21f0931c103d None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:33.843837 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c15a5b0-5781-409d-8a4c-21f0931c103d None admin] PATH_INFO: `/v3/users/glance` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:33.851300 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c15a5b0-5781-409d-8a4c-21f0931c103d None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:33.852749 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c15a5b0-5781-409d-8a4c-21f0931c103d None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:33.853486 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da91088a-776e-4c53-b924-d3c4ccf741fe None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:33.863793 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-0c15a5b0-5781-409d-8a4c-21f0931c103d None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 28 09:31:33.863793 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:33.863793 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:33.863793 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:33.863793 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:33.863793 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:33.863793 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:33.863793 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:33.863793 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:33.863793 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:33.863793 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:33.863793 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:33.863793 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 09:31:33.863793 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 09:31:33.863793 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 09:31:33.863793 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 09:31:33.864818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:33.864818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:33.864818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 09:31:33.864818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:31:33.864818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 09:31:33.864818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:31:33.864818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:33.864818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:33.864818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:33.864818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:33.864818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:33.864818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:33.864818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:33.864818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:33.864818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:33.864818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:33.864818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:33.864818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:33.865837 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:33.865837 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:33.865837 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 09:31:33.865837 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 09:31:33.865837 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 09:31:33.865837 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 09:31:33.865837 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 09:31:33.865837 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 09:31:33.865837 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 28 09:31:33.865837 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:33.865837 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 286/579] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:31:33 2022] GET /identity/v3/users/glance => generated 84 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:33.866982 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c15a5b0-5781-409d-8a4c-21f0931c103d None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:33.892036 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfde84c4-2c0c-48f2-bc82-0be8257070b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:33.892638 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfde84c4-2c0c-48f2-bc82-0be8257070b1 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:33.892916 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfde84c4-2c0c-48f2-bc82-0be8257070b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:33.893192 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfde84c4-2c0c-48f2-bc82-0be8257070b1 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:33.893840 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfde84c4-2c0c-48f2-bc82-0be8257070b1 None admin] RBAC: Adding query filter params (name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:33.894146 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfde84c4-2c0c-48f2-bc82-0be8257070b1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:33.895515 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfde84c4-2c0c-48f2-bc82-0be8257070b1 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:33.903481 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 294/580] 10.222.0.25 () {56 vars in 1103 bytes} [Fri Oct 28 09:31:33 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:33.904283 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfde84c4-2c0c-48f2-bc82-0be8257070b1 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:33.934893 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:33.953671 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:33.968776 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c15a5b0-5781-409d-8a4c-21f0931c103d None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:34.001914 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e68b6545-c1f8-4031-81c5-552cf9078471 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:34.002538 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68b6545-c1f8-4031-81c5-552cf9078471 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:34.002816 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68b6545-c1f8-4031-81c5-552cf9078471 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:34.003088 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68b6545-c1f8-4031-81c5-552cf9078471 None admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:34.003730 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e68b6545-c1f8-4031-81c5-552cf9078471 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:34.004036 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e68b6545-c1f8-4031-81c5-552cf9078471 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:34.004909 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfde84c4-2c0c-48f2-bc82-0be8257070b1 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:34.005405 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e68b6545-c1f8-4031-81c5-552cf9078471 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:34.017617 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 287/581] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:34.018477 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e68b6545-c1f8-4031-81c5-552cf9078471 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:34.040821 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc8ffaee-2c0a-4db2-9c6b-7857f8790f81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:34.041429 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8ffaee-2c0a-4db2-9c6b-7857f8790f81 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:34.041709 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8ffaee-2c0a-4db2-9c6b-7857f8790f81 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:34.041983 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8ffaee-2c0a-4db2-9c6b-7857f8790f81 None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:34.042696 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc8ffaee-2c0a-4db2-9c6b-7857f8790f81 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:34.043000 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc8ffaee-2c0a-4db2-9c6b-7857f8790f81 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:34.044402 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc8ffaee-2c0a-4db2-9c6b-7857f8790f81 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:34.056741 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 295/582] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Oct 28 09:31:33 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:34.057374 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc8ffaee-2c0a-4db2-9c6b-7857f8790f81 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:34.122623 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e68b6545-c1f8-4031-81c5-552cf9078471 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:34.154379 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:34.159260 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704e3909-4682-48fe-bf7a-095af4effceb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:34.159874 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704e3909-4682-48fe-bf7a-095af4effceb None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:34.160152 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704e3909-4682-48fe-bf7a-095af4effceb None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:34.160428 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704e3909-4682-48fe-bf7a-095af4effceb None admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:34.161120 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-704e3909-4682-48fe-bf7a-095af4effceb None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:34.161427 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-704e3909-4682-48fe-bf7a-095af4effceb None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:34.162898 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-704e3909-4682-48fe-bf7a-095af4effceb None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:34.164355 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc8ffaee-2c0a-4db2-9c6b-7857f8790f81 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:34.174556 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 288/583] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Oct 28 09:31:34 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:34.175418 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704e3909-4682-48fe-bf7a-095af4effceb None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:34.185511 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:34.201827 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad540a53-5a87-4f26-9818-ca0d917b0327 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:34.202575 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad540a53-5a87-4f26-9818-ca0d917b0327 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:34.202879 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad540a53-5a87-4f26-9818-ca0d917b0327 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:34.203139 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad540a53-5a87-4f26-9818-ca0d917b0327 None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:34.203868 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad540a53-5a87-4f26-9818-ca0d917b0327 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:34.204175 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad540a53-5a87-4f26-9818-ca0d917b0327 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:34.205598 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad540a53-5a87-4f26-9818-ca0d917b0327 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:34.247250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 296/584] 10.222.0.25 () {56 vars in 1137 bytes} [Fri Oct 28 09:31:34 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:34.248174 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad540a53-5a87-4f26-9818-ca0d917b0327 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:34.282453 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704e3909-4682-48fe-bf7a-095af4effceb None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:34.318261 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133c6803-4e6c-43a0-941a-51b6438b01a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:34.318928 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133c6803-4e6c-43a0-941a-51b6438b01a3 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:34.319207 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133c6803-4e6c-43a0-941a-51b6438b01a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:34.319509 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-133c6803-4e6c-43a0-941a-51b6438b01a3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:34.331265 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-133c6803-4e6c-43a0-941a-51b6438b01a3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:34.332904 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-133c6803-4e6c-43a0-941a-51b6438b01a3 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:34.347833 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-133c6803-4e6c-43a0-941a-51b6438b01a3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:34.347833 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:34.347833 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:34.347833 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:34.347833 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:34.347833 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:34.347833 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:34.347833 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:34.347833 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:34.347833 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:34.347833 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:34.347833 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:34.347833 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:34.347833 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:34.347833 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:34.347833 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:34.348857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:34.348857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:34.348857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:34.348857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:34.348857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:34.348857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:34.348857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:34.348857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:34.348857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:34.348857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:34.348857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:34.348857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:34.348857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:34.348857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:34.348857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:34.348857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:34.348857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:34.349734 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:34.349734 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:34.349734 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:34.349734 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 289/585] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:34.349970 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133c6803-4e6c-43a0-941a-51b6438b01a3 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:34.356632 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad540a53-5a87-4f26-9818-ca0d917b0327 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:34.391582 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a582c201-1c1d-468d-972c-6a85457a5328 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:34.392208 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a582c201-1c1d-468d-972c-6a85457a5328 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:34.392486 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a582c201-1c1d-468d-972c-6a85457a5328 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:34.392761 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a582c201-1c1d-468d-972c-6a85457a5328 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:34.401788 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a582c201-1c1d-468d-972c-6a85457a5328 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:34.403288 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a582c201-1c1d-468d-972c-6a85457a5328 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:34.415694 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:34.415893 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-a582c201-1c1d-468d-972c-6a85457a5328 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 28 09:31:34.415893 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:34.415893 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:34.415893 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:34.415893 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:34.415893 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:34.415893 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:34.415893 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:34.415893 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:34.415893 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:34.415893 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:34.415893 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:34.415893 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 09:31:34.415893 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 09:31:34.415893 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 09:31:34.415893 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 09:31:34.422497 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:34.422497 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:34.422497 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 09:31:34.422497 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:31:34.422497 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 09:31:34.422497 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:31:34.422497 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:34.422497 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:34.422497 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:34.422497 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:34.422497 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:34.422497 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:34.422497 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:34.422497 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:34.422497 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:34.422497 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:34.422497 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:34.422497 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:34.423466 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:34.423466 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:34.423466 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 09:31:34.423466 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 09:31:34.423466 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 09:31:34.423466 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 09:31:34.423466 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 09:31:34.423466 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 09:31:34.423466 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 28 09:31:34.423466 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:34.423466 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 297/586] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:31:34 2022] GET /identity/v3/users/cinder => generated 84 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:34.423466 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a582c201-1c1d-468d-972c-6a85457a5328 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:34.433844 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:34.466856 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-133c6803-4e6c-43a0-941a-51b6438b01a3 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:34.500692 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b33588-b513-44a4-b018-9d47c879d416 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:34.501299 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b33588-b513-44a4-b018-9d47c879d416 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:34.501583 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b33588-b513-44a4-b018-9d47c879d416 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:34.501860 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b33588-b513-44a4-b018-9d47c879d416 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:34.502612 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03b33588-b513-44a4-b018-9d47c879d416 None admin] RBAC: Adding query filter params (role.id=e5d8d090857a4a018d0ec3c79ece6472, scope.project.id=455d1c826b974bc38497d18a516867c7, user.id=9c6cffe0d56e45c4b8fb54fbe5df6094) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:34.502922 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03b33588-b513-44a4-b018-9d47c879d416 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:34.504372 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03b33588-b513-44a4-b018-9d47c879d416 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:34.509733 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 290/587] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:31:34 2022] GET /identity/v3/role_assignments?user.id=9c6cffe0d56e45c4b8fb54fbe5df6094&scope.project.id=455d1c826b974bc38497d18a516867c7&role.id=e5d8d090857a4a018d0ec3c79ece6472 => generated 259 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:34.510536 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b33588-b513-44a4-b018-9d47c879d416 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:34.529981 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a582c201-1c1d-468d-972c-6a85457a5328 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:34.579030 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-763f8b92-53d7-411e-a64a-7ffd0e4b978a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:34.580086 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763f8b92-53d7-411e-a64a-7ffd0e4b978a None admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:34.580373 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763f8b92-53d7-411e-a64a-7ffd0e4b978a None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:34.580797 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763f8b92-53d7-411e-a64a-7ffd0e4b978a None admin] PATH_INFO: `/v3/projects/455d1c826b974bc38497d18a516867c7/users/688ceb201d304c2da5373c3cf36c2d79/roles/e5d8d090857a4a018d0ec3c79ece6472` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:34.601553 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-763f8b92-53d7-411e-a64a-7ffd0e4b978a None admin] RBAC: Authorizing `identity:create_grant(project_id=455d1c826b974bc38497d18a516867c7, user_id=688ceb201d304c2da5373c3cf36c2d79, role_id=e5d8d090857a4a018d0ec3c79ece6472)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:34.603346 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-763f8b92-53d7-411e-a64a-7ffd0e4b978a None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:34.620859 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 298/588] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Oct 28 09:31:34 2022] PUT /identity/v3/projects/455d1c826b974bc38497d18a516867c7/users/688ceb201d304c2da5373c3cf36c2d79/roles/e5d8d090857a4a018d0ec3c79ece6472 => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:31:34.628169 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b33588-b513-44a4-b018-9d47c879d416 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:34.631455 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-763f8b92-53d7-411e-a64a-7ffd0e4b978a None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:34.666321 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:34.688025 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81384959-b347-4476-aa56-f6c10cd0a957 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:34.689031 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81384959-b347-4476-aa56-f6c10cd0a957 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:34.689470 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81384959-b347-4476-aa56-f6c10cd0a957 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:34.689894 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81384959-b347-4476-aa56-f6c10cd0a957 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:34.703331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81384959-b347-4476-aa56-f6c10cd0a957 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:34.705322 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81384959-b347-4476-aa56-f6c10cd0a957 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:34.723640 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-81384959-b347-4476-aa56-f6c10cd0a957 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:34.723640 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:34.723640 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:34.723640 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:34.723640 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:34.723640 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:34.723640 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:34.723640 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:34.723640 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:34.723640 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:34.723640 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:34.723640 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:34.723640 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:34.723640 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:34.723640 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:34.723640 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:34.724972 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:34.724972 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:34.724972 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:34.724972 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:34.724972 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:34.724972 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:34.724972 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:34.724972 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:34.724972 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:34.724972 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:34.724972 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:34.724972 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:34.724972 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:34.724972 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:34.724972 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:34.724972 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:34.724972 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:34.726689 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:34.726689 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:34.726689 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:34.733569 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 291/589] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:34.733569 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81384959-b347-4476-aa56-f6c10cd0a957 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:34.735622 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:34.783150 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-763f8b92-53d7-411e-a64a-7ffd0e4b978a None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:34.825924 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7334ed7-7751-4c50-b4ba-59727e089ce7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:34.826657 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7334ed7-7751-4c50-b4ba-59727e089ce7 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:34.827006 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7334ed7-7751-4c50-b4ba-59727e089ce7 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:34.827371 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7334ed7-7751-4c50-b4ba-59727e089ce7 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:34.828254 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7334ed7-7751-4c50-b4ba-59727e089ce7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:34.828634 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7334ed7-7751-4c50-b4ba-59727e089ce7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:34.830112 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7334ed7-7751-4c50-b4ba-59727e089ce7 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:34.843696 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 299/590] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:34.844637 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7334ed7-7751-4c50-b4ba-59727e089ce7 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:34.860724 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81384959-b347-4476-aa56-f6c10cd0a957 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:34.897300 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5337fdf8-a8c5-4959-9dde-cf86a03bdaf4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:34.898749 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5337fdf8-a8c5-4959-9dde-cf86a03bdaf4 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:34.898900 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5337fdf8-a8c5-4959-9dde-cf86a03bdaf4 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:34.899134 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5337fdf8-a8c5-4959-9dde-cf86a03bdaf4 None admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:34.899873 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5337fdf8-a8c5-4959-9dde-cf86a03bdaf4 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:34.900181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5337fdf8-a8c5-4959-9dde-cf86a03bdaf4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:34.901616 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5337fdf8-a8c5-4959-9dde-cf86a03bdaf4 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:34.924017 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 292/591] 10.222.0.25 () {56 vars in 1137 bytes} [Fri Oct 28 09:31:34 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:34.924899 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5337fdf8-a8c5-4959-9dde-cf86a03bdaf4 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:34.968769 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:34.971355 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7334ed7-7751-4c50-b4ba-59727e089ce7 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:34.992833 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:35.016501 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0521b9cc-1ee1-42c4-9e13-76732a5be059 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:35.017126 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0521b9cc-1ee1-42c4-9e13-76732a5be059 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:35.017410 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0521b9cc-1ee1-42c4-9e13-76732a5be059 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:35.017688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0521b9cc-1ee1-42c4-9e13-76732a5be059 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:35.018337 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0521b9cc-1ee1-42c4-9e13-76732a5be059 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:35.018713 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0521b9cc-1ee1-42c4-9e13-76732a5be059 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:35.020139 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0521b9cc-1ee1-42c4-9e13-76732a5be059 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:35.040966 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 300/592] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:35.040966 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0521b9cc-1ee1-42c4-9e13-76732a5be059 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:35.057693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5337fdf8-a8c5-4959-9dde-cf86a03bdaf4 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:35.108737 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3a538d-7d4f-4e70-ab66-f13a8701db85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:35.109750 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3a538d-7d4f-4e70-ab66-f13a8701db85 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:35.110350 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3a538d-7d4f-4e70-ab66-f13a8701db85 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:35.110893 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3a538d-7d4f-4e70-ab66-f13a8701db85 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:35.120372 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3a538d-7d4f-4e70-ab66-f13a8701db85 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:35.122545 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3a538d-7d4f-4e70-ab66-f13a8701db85 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:35.138630 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-ff3a538d-7d4f-4e70-ab66-f13a8701db85 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 28 09:31:35.138630 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:35.138630 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:35.138630 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:35.138630 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:35.138630 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:35.138630 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:35.138630 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:35.138630 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:35.138630 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:35.138630 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:35.138630 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:35.138630 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 09:31:35.138630 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 09:31:35.138630 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 09:31:35.138630 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 09:31:35.139729 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:35.139729 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:35.139729 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 09:31:35.139729 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:31:35.139729 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 09:31:35.139729 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:31:35.139729 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:35.139729 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:35.139729 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:35.139729 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:35.139729 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:35.139729 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:35.139729 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:35.139729 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:35.139729 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:35.139729 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:35.139729 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:35.139729 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:35.140803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:35.140803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:35.140803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 09:31:35.140803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 09:31:35.140803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 09:31:35.140803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 09:31:35.140803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 09:31:35.140803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 09:31:35.140803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 28 09:31:35.140803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:35.143285 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 293/593] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Oct 28 09:31:34 2022] GET /identity/v3/users/neutron => generated 85 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:35.144646 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3a538d-7d4f-4e70-ab66-f13a8701db85 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:35.195739 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0521b9cc-1ee1-42c4-9e13-76732a5be059 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:35.234157 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b336b14-2003-4e56-bb5a-5e36238f803d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:35.235042 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b336b14-2003-4e56-bb5a-5e36238f803d None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:35.235472 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b336b14-2003-4e56-bb5a-5e36238f803d None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:35.235851 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b336b14-2003-4e56-bb5a-5e36238f803d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:35.247361 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b336b14-2003-4e56-bb5a-5e36238f803d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:35.248977 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b336b14-2003-4e56-bb5a-5e36238f803d None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:35.265977 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-4b336b14-2003-4e56-bb5a-5e36238f803d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:35.265977 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:35.265977 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:35.265977 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:35.265977 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:35.265977 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:35.265977 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:35.265977 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:35.265977 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:35.265977 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:35.265977 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:35.265977 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:35.265977 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:35.265977 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:35.265977 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:35.265977 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:35.267214 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:35.267214 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:35.267214 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:35.267214 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:35.267214 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:35.267214 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:35.267214 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:35.267214 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:35.267214 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:35.267214 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:35.267214 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:35.267214 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:35.267214 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:35.267214 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:35.267214 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:35.267214 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:35.267214 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:35.268293 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:35.268293 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:35.268293 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:35.269855 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 301/594] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:35.270708 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b336b14-2003-4e56-bb5a-5e36238f803d None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:35.278146 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:35.293864 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3a538d-7d4f-4e70-ab66-f13a8701db85 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:35.311713 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:35.337738 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063430a8-4ecf-4902-b6b0-06496f496c0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:35.339005 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063430a8-4ecf-4902-b6b0-06496f496c0f None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:35.339439 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063430a8-4ecf-4902-b6b0-06496f496c0f None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:35.340169 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063430a8-4ecf-4902-b6b0-06496f496c0f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:35.351180 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-063430a8-4ecf-4902-b6b0-06496f496c0f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:35.353879 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-063430a8-4ecf-4902-b6b0-06496f496c0f None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:35.373447 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-063430a8-4ecf-4902-b6b0-06496f496c0f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:31:35.373447 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:35.373447 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:35.373447 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:35.373447 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:35.373447 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:35.373447 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:35.373447 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:35.373447 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:35.373447 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:35.373447 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:35.373447 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:35.373447 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 09:31:35.373447 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 09:31:35.373447 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 09:31:35.374568 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 09:31:35.374568 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:35.374568 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:35.374568 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:35.374568 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:35.374568 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:35.374568 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:35.374568 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:35.374568 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:35.374568 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:35.374568 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:35.374568 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:35.374568 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:35.374568 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:35.374568 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:35.374568 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:35.374568 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:35.375668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 09:31:35.375668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 09:31:35.375668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 09:31:35.375668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 09:31:35.375668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 09:31:35.375668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 09:31:35.375668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:31:35.375668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:35.378667 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 294/595] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:31:35 2022] GET /identity/v3/projects/service => generated 88 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:35.379800 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063430a8-4ecf-4902-b6b0-06496f496c0f None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:35.412906 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b336b14-2003-4e56-bb5a-5e36238f803d None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:35.457442 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9ab219-4795-43d4-9a48-b99073d360af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:35.458217 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9ab219-4795-43d4-9a48-b99073d360af None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:35.458652 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9ab219-4795-43d4-9a48-b99073d360af None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:35.459051 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9ab219-4795-43d4-9a48-b99073d360af None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:35.459925 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c9ab219-4795-43d4-9a48-b99073d360af None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:35.460876 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c9ab219-4795-43d4-9a48-b99073d360af None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:35.462875 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c9ab219-4795-43d4-9a48-b99073d360af None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:35.483626 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 302/596] 10.222.0.25 () {56 vars in 1139 bytes} [Fri Oct 28 09:31:35 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:35.484841 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9ab219-4795-43d4-9a48-b99073d360af None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:35.530521 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063430a8-4ecf-4902-b6b0-06496f496c0f None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:35.576363 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7923f0d-db0b-4df6-a218-2d59d5a09776 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:35.577168 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7923f0d-db0b-4df6-a218-2d59d5a09776 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:35.577554 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7923f0d-db0b-4df6-a218-2d59d5a09776 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:35.577930 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7923f0d-db0b-4df6-a218-2d59d5a09776 None admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:35.578893 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7923f0d-db0b-4df6-a218-2d59d5a09776 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:35.579127 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7923f0d-db0b-4df6-a218-2d59d5a09776 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:35.580702 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7923f0d-db0b-4df6-a218-2d59d5a09776 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:35.594973 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 295/597] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:35.596072 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7923f0d-db0b-4df6-a218-2d59d5a09776 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:35.606751 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:35.614070 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:35.625867 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9ab219-4795-43d4-9a48-b99073d360af None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:35.674120 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad3f7784-143f-43ed-84e8-257b039d498d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:35.675140 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3f7784-143f-43ed-84e8-257b039d498d None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:35.675522 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3f7784-143f-43ed-84e8-257b039d498d None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:35.675906 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3f7784-143f-43ed-84e8-257b039d498d None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:35.676753 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad3f7784-143f-43ed-84e8-257b039d498d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:35.677162 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad3f7784-143f-43ed-84e8-257b039d498d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:35.679377 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad3f7784-143f-43ed-84e8-257b039d498d None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:35.698234 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 303/598] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Oct 28 09:31:35 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:35.699586 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad3f7784-143f-43ed-84e8-257b039d498d None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:35.722499 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7923f0d-db0b-4df6-a218-2d59d5a09776 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:35.769556 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a08776-ba2f-4490-a21e-8b6ff76a96ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:35.770922 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a08776-ba2f-4490-a21e-8b6ff76a96ff None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:35.771509 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a08776-ba2f-4490-a21e-8b6ff76a96ff None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:35.772032 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a08776-ba2f-4490-a21e-8b6ff76a96ff None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:35.786046 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16a08776-ba2f-4490-a21e-8b6ff76a96ff None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:35.788788 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16a08776-ba2f-4490-a21e-8b6ff76a96ff None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:35.809369 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-16a08776-ba2f-4490-a21e-8b6ff76a96ff None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:35.809369 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:35.809369 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:35.809369 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:35.809369 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:35.809369 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:35.809369 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:35.809369 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:35.809369 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:35.809369 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:35.809369 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:35.809369 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:35.809369 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:35.809369 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:35.809369 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:35.809369 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:35.810519 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:35.810519 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:35.810519 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:35.810519 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:35.810519 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:35.810519 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:35.810519 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:35.810519 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:35.810519 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:35.810519 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:35.810519 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:35.810519 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:35.810519 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:35.810519 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:35.810519 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:35.810519 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:35.810519 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:35.811552 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:35.811552 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:35.811552 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:35.814152 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 296/599] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:35.815108 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a08776-ba2f-4490-a21e-8b6ff76a96ff None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:35.821064 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad3f7784-143f-43ed-84e8-257b039d498d None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:35.861095 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b31297-3801-4277-a3d6-f3f69cfe46c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:35.862063 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b31297-3801-4277-a3d6-f3f69cfe46c2 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:35.863451 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b31297-3801-4277-a3d6-f3f69cfe46c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:35.863451 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b31297-3801-4277-a3d6-f3f69cfe46c2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:35.873937 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7b31297-3801-4277-a3d6-f3f69cfe46c2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:35.875726 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7b31297-3801-4277-a3d6-f3f69cfe46c2 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:35.889659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-a7b31297-3801-4277-a3d6-f3f69cfe46c2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:31:35.889659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:35.889659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:35.889659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:35.889659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:35.889659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:35.889659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:35.889659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:35.889659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:35.889659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:35.889659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:35.889659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:35.889659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 09:31:35.889659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 09:31:35.889659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 09:31:35.890732 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 09:31:35.890732 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:35.890732 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:35.890732 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:35.890732 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:35.890732 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:35.890732 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:35.890732 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:35.890732 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:35.890732 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:35.890732 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:35.890732 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:35.890732 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:35.890732 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:35.890732 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:35.890732 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:35.890732 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:35.891749 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 09:31:35.891749 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 09:31:35.891749 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 09:31:35.891749 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 09:31:35.891749 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 09:31:35.891749 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 09:31:35.891749 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:31:35.891749 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:35.893659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 304/600] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:31:35 2022] GET /identity/v3/projects/service => generated 88 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:35.894716 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b31297-3801-4277-a3d6-f3f69cfe46c2 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:35.901209 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:35.910840 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:35.954631 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a08776-ba2f-4490-a21e-8b6ff76a96ff None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:36.000046 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896dab86-458b-443d-adf9-ee18c8aeada8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:36.001376 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896dab86-458b-443d-adf9-ee18c8aeada8 None admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:36.001806 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896dab86-458b-443d-adf9-ee18c8aeada8 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:36.002353 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896dab86-458b-443d-adf9-ee18c8aeada8 None admin] PATH_INFO: `/v3/projects/455d1c826b974bc38497d18a516867c7/users/f4b2ea9aa31445d69e0c1d96b5657708/roles/e5d8d090857a4a018d0ec3c79ece6472` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:36.017548 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b31297-3801-4277-a3d6-f3f69cfe46c2 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:36.023309 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-896dab86-458b-443d-adf9-ee18c8aeada8 None admin] RBAC: Authorizing `identity:create_grant(project_id=455d1c826b974bc38497d18a516867c7, user_id=f4b2ea9aa31445d69e0c1d96b5657708, role_id=e5d8d090857a4a018d0ec3c79ece6472)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:36.025035 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-896dab86-458b-443d-adf9-ee18c8aeada8 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:36.045435 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 297/601] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Oct 28 09:31:35 2022] PUT /identity/v3/projects/455d1c826b974bc38497d18a516867c7/users/f4b2ea9aa31445d69e0c1d96b5657708/roles/e5d8d090857a4a018d0ec3c79ece6472 => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:31:36.046958 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896dab86-458b-443d-adf9-ee18c8aeada8 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:36.056532 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d82315d-88fa-46fb-8a77-8a30e5bde270 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:36.057158 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d82315d-88fa-46fb-8a77-8a30e5bde270 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:36.057451 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d82315d-88fa-46fb-8a77-8a30e5bde270 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:36.057716 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d82315d-88fa-46fb-8a77-8a30e5bde270 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:36.058378 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d82315d-88fa-46fb-8a77-8a30e5bde270 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:36.058729 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d82315d-88fa-46fb-8a77-8a30e5bde270 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:36.060663 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d82315d-88fa-46fb-8a77-8a30e5bde270 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:36.074662 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 305/602] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:36.080548 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d82315d-88fa-46fb-8a77-8a30e5bde270 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:36.167775 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:36.214959 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d82315d-88fa-46fb-8a77-8a30e5bde270 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:36.239460 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896dab86-458b-443d-adf9-ee18c8aeada8 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:36.260522 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9cf651-5152-4e73-a01d-9f1d918bf3a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:36.261547 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9cf651-5152-4e73-a01d-9f1d918bf3a1 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:36.262058 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9cf651-5152-4e73-a01d-9f1d918bf3a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:36.262568 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d9cf651-5152-4e73-a01d-9f1d918bf3a1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:36.271599 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d9cf651-5152-4e73-a01d-9f1d918bf3a1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:36.274004 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d9cf651-5152-4e73-a01d-9f1d918bf3a1 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:36.282845 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:36.289021 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-1d9cf651-5152-4e73-a01d-9f1d918bf3a1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:31:36.289021 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:36.289021 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:36.289021 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:36.289021 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:36.289021 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:36.289021 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:36.289021 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:36.289021 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:36.289021 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:36.289021 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:36.289021 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:36.289021 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 09:31:36.289021 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 09:31:36.289021 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 09:31:36.289981 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 09:31:36.289981 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:36.289981 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:36.289981 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:36.289981 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:36.289981 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:36.289981 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:36.289981 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:36.289981 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:36.289981 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:36.289981 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:36.289981 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:36.289981 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:36.289981 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:36.289981 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:36.289981 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:36.289981 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:36.291467 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 09:31:36.291467 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 09:31:36.291467 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 09:31:36.291467 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 09:31:36.291467 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 09:31:36.291467 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 09:31:36.291467 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:31:36.291467 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:36.291467 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 306/603] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:31:36 2022] GET /identity/v3/projects/service => generated 88 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:36.297382 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9cf651-5152-4e73-a01d-9f1d918bf3a1 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:36.324785 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff81cac-0c4f-4fbe-a7ba-857937dd6d18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:36.325467 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff81cac-0c4f-4fbe-a7ba-857937dd6d18 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:36.325692 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff81cac-0c4f-4fbe-a7ba-857937dd6d18 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:36.325965 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff81cac-0c4f-4fbe-a7ba-857937dd6d18 None admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:36.326917 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ff81cac-0c4f-4fbe-a7ba-857937dd6d18 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:36.327321 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ff81cac-0c4f-4fbe-a7ba-857937dd6d18 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:36.328798 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ff81cac-0c4f-4fbe-a7ba-857937dd6d18 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:36.346043 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 298/604] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Oct 28 09:31:36 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 300 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:36.352139 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff81cac-0c4f-4fbe-a7ba-857937dd6d18 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:36.464112 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d9cf651-5152-4e73-a01d-9f1d918bf3a1 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:36.496436 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff81cac-0c4f-4fbe-a7ba-857937dd6d18 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:36.522136 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:36.547254 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb14e0aa-cd3c-43fd-98c6-027392845d38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:36.548263 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb14e0aa-cd3c-43fd-98c6-027392845d38 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:36.548798 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb14e0aa-cd3c-43fd-98c6-027392845d38 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:36.551476 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb14e0aa-cd3c-43fd-98c6-027392845d38 None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:36.552491 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb14e0aa-cd3c-43fd-98c6-027392845d38 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:36.554077 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb14e0aa-cd3c-43fd-98c6-027392845d38 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:36.556609 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb14e0aa-cd3c-43fd-98c6-027392845d38 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:36.579943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c43ff8-4d25-489d-b77d-834f78324ddf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:36.582089 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c43ff8-4d25-489d-b77d-834f78324ddf None admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:36.582378 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c43ff8-4d25-489d-b77d-834f78324ddf None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:36.582679 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c43ff8-4d25-489d-b77d-834f78324ddf None admin] PATH_INFO: `/v3/projects/455d1c826b974bc38497d18a516867c7/users/1021f1cc73b54d6eaa647a5f49815ef8/roles/e5d8d090857a4a018d0ec3c79ece6472` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:36.585839 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 307/605] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Oct 28 09:31:36 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:31:36.591771 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb14e0aa-cd3c-43fd-98c6-027392845d38 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:36.617405 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c43ff8-4d25-489d-b77d-834f78324ddf None admin] RBAC: Authorizing `identity:create_grant(project_id=455d1c826b974bc38497d18a516867c7, user_id=1021f1cc73b54d6eaa647a5f49815ef8, role_id=e5d8d090857a4a018d0ec3c79ece6472)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:36.618917 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c43ff8-4d25-489d-b77d-834f78324ddf None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:36.649627 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 299/606] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Oct 28 09:31:36 2022] PUT /identity/v3/projects/455d1c826b974bc38497d18a516867c7/users/1021f1cc73b54d6eaa647a5f49815ef8/roles/e5d8d090857a4a018d0ec3c79ece6472 => generated 0 bytes in 298 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:31:36.752622 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb14e0aa-cd3c-43fd-98c6-027392845d38 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:36.801142 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce29e8fb-ce3c-4fdb-a852-e3660f1ed3d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:36.802271 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce29e8fb-ce3c-4fdb-a852-e3660f1ed3d7 None admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:36.802810 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce29e8fb-ce3c-4fdb-a852-e3660f1ed3d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:36.803320 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce29e8fb-ce3c-4fdb-a852-e3660f1ed3d7 None admin] PATH_INFO: `/v3/projects/455d1c826b974bc38497d18a516867c7/users/1db6b98e87e64584932447c82fefc3b7/roles/e5d8d090857a4a018d0ec3c79ece6472` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:36.829673 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce29e8fb-ce3c-4fdb-a852-e3660f1ed3d7 None admin] RBAC: Authorizing `identity:create_grant(project_id=455d1c826b974bc38497d18a516867c7, user_id=1db6b98e87e64584932447c82fefc3b7, role_id=e5d8d090857a4a018d0ec3c79ece6472)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:36.831976 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce29e8fb-ce3c-4fdb-a852-e3660f1ed3d7 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:36.843905 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce51d0db-a643-443f-9db1-c650b8ccccbf None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:36.844279 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce51d0db-a643-443f-9db1-c650b8ccccbf None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:36.844597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce51d0db-a643-443f-9db1-c650b8ccccbf None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:36.847657 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 300/607] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:36.853033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca34b42a-a4be-40ba-abc9-224aa053043e None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:36.853370 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca34b42a-a4be-40ba-abc9-224aa053043e None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:36.853661 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca34b42a-a4be-40ba-abc9-224aa053043e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:36.856147 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 308/608] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Oct 28 09:31:36 2022] PUT /identity/v3/projects/455d1c826b974bc38497d18a516867c7/users/1db6b98e87e64584932447c82fefc3b7/roles/e5d8d090857a4a018d0ec3c79ece6472 => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:31:36.885790 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-ca34b42a-a4be-40ba-abc9-224aa053043e None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:36.887051 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-ca34b42a-a4be-40ba-abc9-224aa053043e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:36.915464 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:36.985550 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 301/609] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 09:31:37.129065 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea520329-e653-4a2b-98a7-c20e752a0a52 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:37.129852 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea520329-e653-4a2b-98a7-c20e752a0a52 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:37.130341 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea520329-e653-4a2b-98a7-c20e752a0a52 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:37.132311 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 309/610] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:37.137751 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e1a30a-c672-4c91-b3a2-42415e6bb97a None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:37.138083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e1a30a-c672-4c91-b3a2-42415e6bb97a None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:37.138604 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e1a30a-c672-4c91-b3a2-42415e6bb97a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:37.167852 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-58e1a30a-c672-4c91-b3a2-42415e6bb97a None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:37.169695 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-58e1a30a-c672-4c91-b3a2-42415e6bb97a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:37.279572 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 302/611] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:31:37.349536 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea520329-e653-4a2b-98a7-c20e752a0a52 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:37.397272 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:37.462203 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea520329-e653-4a2b-98a7-c20e752a0a52 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:37.500079 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b79945-6327-425d-8086-b074189e3e8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:37.500722 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b79945-6327-425d-8086-b074189e3e8f None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:37.501277 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b79945-6327-425d-8086-b074189e3e8f None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:37.501549 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b79945-6327-425d-8086-b074189e3e8f None admin] PATH_INFO: `/v3/roles/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:37.512830 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95b79945-6327-425d-8086-b074189e3e8f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:31:37.517701 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-95b79945-6327-425d-8086-b074189e3e8f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 09:31:37.517701 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:37.517701 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:37.517701 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:37.517701 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:37.517701 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:37.517701 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:37.517701 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:37.517701 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:37.517701 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:37.517701 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:37.517701 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:37.517701 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:31:37.517701 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:31:37.517701 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:31:37.517701 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:31:37.518637 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:31:37.518637 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:31:37.518637 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:31:37.518637 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:31:37.518637 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:37.518637 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:37.518637 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:37.518637 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:37.518637 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:37.518637 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:37.518637 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:37.518637 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:37.518637 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:37.518637 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:37.518637 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:37.518637 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:37.519464 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:37.519464 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:37.519464 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:37.519464 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:37.519464 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:31:37.519464 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:31:37.519464 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:31:37.519464 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:31:37.519464 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:31:37.519464 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:31:37.519464 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 09:31:37.519464 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:37.519464 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 310/612] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Oct 28 09:31:37 2022] GET /identity/v3/roles/service => generated 85 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:37.526142 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e1a30a-c672-4c91-b3a2-42415e6bb97a None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:37.642270 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b79945-6327-425d-8086-b074189e3e8f None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:37.678498 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:37.704232 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e1a30a-c672-4c91-b3a2-42415e6bb97a None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:37.762525 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b79945-6327-425d-8086-b074189e3e8f None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:37.799680 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fe2cfe-fc9b-46ae-b32c-e66c24b484c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:37.799943 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3724958c-5eaa-4c6e-92cd-bee38372e845 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:37.800314 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fe2cfe-fc9b-46ae-b32c-e66c24b484c0 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:37.800421 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3724958c-5eaa-4c6e-92cd-bee38372e845 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:37.800602 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fe2cfe-fc9b-46ae-b32c-e66c24b484c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:37.800708 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3724958c-5eaa-4c6e-92cd-bee38372e845 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:37.800877 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fe2cfe-fc9b-46ae-b32c-e66c24b484c0 None admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:37.800966 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3724958c-5eaa-4c6e-92cd-bee38372e845 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:37.801553 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62fe2cfe-fc9b-46ae-b32c-e66c24b484c0 None admin] RBAC: Adding query filter params (name=service) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:37.801862 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62fe2cfe-fc9b-46ae-b32c-e66c24b484c0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:37.803298 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62fe2cfe-fc9b-46ae-b32c-e66c24b484c0 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:37.804324 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:37.813835 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3724958c-5eaa-4c6e-92cd-bee38372e845 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:31:37.818910 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-3724958c-5eaa-4c6e-92cd-bee38372e845 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 09:31:37.818910 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:37.818910 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:37.818910 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:37.818910 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:37.818910 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:37.818910 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:37.818910 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:37.818910 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:37.818910 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:37.818910 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:37.818910 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:37.818910 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:31:37.818910 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:31:37.818910 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:31:37.818910 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:31:37.819916 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:31:37.819916 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:31:37.819916 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:31:37.819916 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:31:37.819916 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:37.819916 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:37.819916 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:37.819916 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:37.819916 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:37.819916 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:37.819916 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:37.819916 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:37.819916 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:37.819916 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:37.819916 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:37.819916 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:37.820766 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:37.820766 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:37.820766 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:37.820766 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:37.820766 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:31:37.820766 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:31:37.820766 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:31:37.820766 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:31:37.820766 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:31:37.820766 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:31:37.820766 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 09:31:37.820766 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:37.820766 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 311/613] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Oct 28 09:31:37 2022] GET /identity/v3/roles/service => generated 85 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:37.826653 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 303/614] 10.222.0.25 () {56 vars in 1103 bytes} [Fri Oct 28 09:31:37 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 301 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:37.828052 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3724958c-5eaa-4c6e-92cd-bee38372e845 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:37.832098 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fe2cfe-fc9b-46ae-b32c-e66c24b484c0 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:37.967695 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3724958c-5eaa-4c6e-92cd-bee38372e845 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:38.005484 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff843ee-5280-4a58-8c4e-2c2a2ab3bbae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:38.006102 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff843ee-5280-4a58-8c4e-2c2a2ab3bbae None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:38.006386 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff843ee-5280-4a58-8c4e-2c2a2ab3bbae None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:38.006706 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff843ee-5280-4a58-8c4e-2c2a2ab3bbae None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:38.007935 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff843ee-5280-4a58-8c4e-2c2a2ab3bbae None admin] RBAC: Adding query filter params (name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:38.008272 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff843ee-5280-4a58-8c4e-2c2a2ab3bbae None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:38.009855 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff843ee-5280-4a58-8c4e-2c2a2ab3bbae None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:38.019663 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 312/615] 10.222.0.25 () {56 vars in 1103 bytes} [Fri Oct 28 09:31:37 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:31:38.025811 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff843ee-5280-4a58-8c4e-2c2a2ab3bbae None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:38.042350 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fe2cfe-fc9b-46ae-b32c-e66c24b484c0 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:38.076986 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:38.103730 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda38cf1-5464-44fd-a121-1629c77d79e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:38.104381 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda38cf1-5464-44fd-a121-1629c77d79e0 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:38.104657 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda38cf1-5464-44fd-a121-1629c77d79e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:38.104927 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda38cf1-5464-44fd-a121-1629c77d79e0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:38.126356 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eda38cf1-5464-44fd-a121-1629c77d79e0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:38.131150 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eda38cf1-5464-44fd-a121-1629c77d79e0 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:38.143779 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:38.154133 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-eda38cf1-5464-44fd-a121-1629c77d79e0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:38.154133 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:38.154133 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:38.154133 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:38.154133 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:38.154133 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:38.154133 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:38.154133 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:38.154133 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:38.154133 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:38.154133 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:38.154133 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:38.154133 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:38.154133 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:38.154133 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:38.154133 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:38.155105 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:38.155105 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:38.155105 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:38.155105 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:38.155105 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:38.155105 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:38.155105 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:38.155105 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:38.155105 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:38.155105 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:38.155105 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:38.155105 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:38.155105 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:38.155105 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:38.155105 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:38.155105 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:38.155105 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:38.155998 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:38.155998 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:38.155998 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:38.155998 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 304/616] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 324 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:38.161986 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda38cf1-5464-44fd-a121-1629c77d79e0 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:38.226688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff843ee-5280-4a58-8c4e-2c2a2ab3bbae None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:38.288961 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70ed95a-ccf0-4d26-a246-17d14eb325c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:38.289603 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70ed95a-ccf0-4d26-a246-17d14eb325c7 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:38.289886 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70ed95a-ccf0-4d26-a246-17d14eb325c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:38.290166 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70ed95a-ccf0-4d26-a246-17d14eb325c7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:38.301787 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d70ed95a-ccf0-4d26-a246-17d14eb325c7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:38.303275 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d70ed95a-ccf0-4d26-a246-17d14eb325c7 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:38.329806 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-d70ed95a-ccf0-4d26-a246-17d14eb325c7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:38.329806 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:38.329806 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:38.329806 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:38.329806 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:38.329806 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:38.329806 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:38.329806 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:38.329806 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:38.329806 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:38.329806 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:38.329806 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:38.329806 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:38.329806 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:38.329806 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:38.329806 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:38.330845 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:38.330845 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:38.330845 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:38.330845 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:38.330845 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:38.330845 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:38.330845 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:38.330845 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:38.330845 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:38.330845 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:38.330845 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:38.330845 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:38.330845 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:38.330845 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:38.330845 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:38.330845 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:38.330845 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:38.332676 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:38.332676 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:38.332676 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:38.332676 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 313/617] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 306 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:38.340375 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70ed95a-ccf0-4d26-a246-17d14eb325c7 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:38.361372 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda38cf1-5464-44fd-a121-1629c77d79e0 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:38.435445 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f096427f-44bb-4ff2-b96d-27a0a1119067 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:38.436484 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f096427f-44bb-4ff2-b96d-27a0a1119067 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:38.436959 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f096427f-44bb-4ff2-b96d-27a0a1119067 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:38.437426 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f096427f-44bb-4ff2-b96d-27a0a1119067 None admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:38.438275 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f096427f-44bb-4ff2-b96d-27a0a1119067 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:38.438794 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f096427f-44bb-4ff2-b96d-27a0a1119067 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:38.440389 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f096427f-44bb-4ff2-b96d-27a0a1119067 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:38.452944 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:38.469688 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 305/618] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 308 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:38.474749 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f096427f-44bb-4ff2-b96d-27a0a1119067 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:38.491910 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70ed95a-ccf0-4d26-a246-17d14eb325c7 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:38.530181 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6612676e-0298-40cf-a54a-d8f94776de55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:38.530788 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6612676e-0298-40cf-a54a-d8f94776de55 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:38.530990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6612676e-0298-40cf-a54a-d8f94776de55 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:38.531570 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6612676e-0298-40cf-a54a-d8f94776de55 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:38.532223 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6612676e-0298-40cf-a54a-d8f94776de55 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:38.533171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6612676e-0298-40cf-a54a-d8f94776de55 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:38.534551 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6612676e-0298-40cf-a54a-d8f94776de55 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:38.548519 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 314/619] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:38.554471 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6612676e-0298-40cf-a54a-d8f94776de55 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:38.624294 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:38.680460 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6612676e-0298-40cf-a54a-d8f94776de55 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:38.716194 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:38.719795 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f096427f-44bb-4ff2-b96d-27a0a1119067 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:38.723039 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbbde77c-8dfe-40f0-8b46-ff2938e1ba0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:38.723821 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbbde77c-8dfe-40f0-8b46-ff2938e1ba0b None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:38.724105 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbbde77c-8dfe-40f0-8b46-ff2938e1ba0b None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:38.724383 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbbde77c-8dfe-40f0-8b46-ff2938e1ba0b None admin] PATH_INFO: `/v3/users/nova` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:38.736929 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbbde77c-8dfe-40f0-8b46-ff2938e1ba0b None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:38.739003 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbbde77c-8dfe-40f0-8b46-ff2938e1ba0b None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:38.754184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-dbbde77c-8dfe-40f0-8b46-ff2938e1ba0b None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 28 09:31:38.754184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:38.754184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:38.754184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:38.754184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:38.754184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:38.754184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:38.754184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:38.754184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:38.754184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:38.754184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:38.754184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:38.754184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 09:31:38.754184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 09:31:38.754184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 09:31:38.754184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 09:31:38.755230 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:38.755230 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:38.755230 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 09:31:38.755230 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:31:38.755230 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 09:31:38.755230 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:31:38.755230 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:38.755230 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:38.755230 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:38.755230 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:38.755230 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:38.755230 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:38.755230 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:38.755230 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:38.755230 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:38.755230 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:38.755230 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:38.755230 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:38.756187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:38.756187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:38.756187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 09:31:38.756187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 09:31:38.756187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 09:31:38.756187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 09:31:38.756187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 09:31:38.756187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 09:31:38.756187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 28 09:31:38.756187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:38.756187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 315/620] 10.222.0.25 () {56 vars in 1093 bytes} [Fri Oct 28 09:31:38 2022] GET /identity/v3/users/nova => generated 82 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:38.763219 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbbde77c-8dfe-40f0-8b46-ff2938e1ba0b None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:38.799013 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb9cef9-a80c-4905-a140-aa2db534ea3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:38.800611 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb9cef9-a80c-4905-a140-aa2db534ea3e None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:38.801110 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb9cef9-a80c-4905-a140-aa2db534ea3e None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:38.801848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb9cef9-a80c-4905-a140-aa2db534ea3e None admin] PATH_INFO: `/v3/users/swift` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:38.812759 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb9cef9-a80c-4905-a140-aa2db534ea3e None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:38.814791 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb9cef9-a80c-4905-a140-aa2db534ea3e None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:38.832803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-ceb9cef9-a80c-4905-a140-aa2db534ea3e None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 28 09:31:38.832803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:38.832803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:38.832803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:38.832803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:38.832803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:38.832803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:38.832803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:38.832803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:38.832803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:38.832803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:38.832803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:38.832803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 09:31:38.832803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 09:31:38.832803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 09:31:38.832803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 09:31:38.833809 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:38.833809 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:38.833809 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 09:31:38.833809 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:31:38.833809 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 09:31:38.833809 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:31:38.833809 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:38.833809 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:38.833809 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:38.833809 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:38.833809 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:38.833809 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:38.833809 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:38.833809 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:38.833809 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:38.833809 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:38.833809 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:38.833809 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:38.834799 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:38.834799 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:38.834799 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 09:31:38.834799 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 09:31:38.834799 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 09:31:38.834799 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 09:31:38.834799 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 09:31:38.834799 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 09:31:38.834799 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 28 09:31:38.834799 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:38.838004 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 306/621] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Oct 28 09:31:38 2022] GET /identity/v3/users/swift => generated 83 bytes in 364 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:31:38.840190 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e095fb1-b8c0-4d10-b372-f9502deea890 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:38.840634 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e095fb1-b8c0-4d10-b372-f9502deea890 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:38.841333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e095fb1-b8c0-4d10-b372-f9502deea890 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:38.847368 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 307/622] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:38 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:38.847543 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e095fb1-b8c0-4d10-b372-f9502deea890 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:38.896974 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbbde77c-8dfe-40f0-8b46-ff2938e1ba0b None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:38.940721 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aefa24e-2c74-4f49-8a21-e5067bf1b14a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:38.941340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aefa24e-2c74-4f49-8a21-e5067bf1b14a None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:38.941623 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aefa24e-2c74-4f49-8a21-e5067bf1b14a None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:38.941899 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aefa24e-2c74-4f49-8a21-e5067bf1b14a None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:38.942678 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aefa24e-2c74-4f49-8a21-e5067bf1b14a None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:38.942987 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aefa24e-2c74-4f49-8a21-e5067bf1b14a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:38.944945 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aefa24e-2c74-4f49-8a21-e5067bf1b14a None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:38.969367 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 316/623] 10.222.0.25 () {56 vars in 1133 bytes} [Fri Oct 28 09:31:38 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:38.970818 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5350240c-ab31-40e4-bb04-a8d15bf46bec None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:38.971114 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5350240c-ab31-40e4-bb04-a8d15bf46bec None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:38.971397 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5350240c-ab31-40e4-bb04-a8d15bf46bec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:38.991138 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-5350240c-ab31-40e4-bb04-a8d15bf46bec None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:38.991996 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e095fb1-b8c0-4d10-b372-f9502deea890 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:38.992392 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-5350240c-ab31-40e4-bb04-a8d15bf46bec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:39.016300 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:39.040826 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd63503-3552-4392-8e0c-403a900d43cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:39.041816 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd63503-3552-4392-8e0c-403a900d43cb None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:39.042772 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd63503-3552-4392-8e0c-403a900d43cb None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:39.043283 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd63503-3552-4392-8e0c-403a900d43cb None admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:39.044824 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd63503-3552-4392-8e0c-403a900d43cb None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:39.045420 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd63503-3552-4392-8e0c-403a900d43cb None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:39.047188 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:39.047766 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd63503-3552-4392-8e0c-403a900d43cb None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:39.071564 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 308/624] 10.222.0.25 () {56 vars in 1135 bytes} [Fri Oct 28 09:31:38 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:39.072665 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd63503-3552-4392-8e0c-403a900d43cb None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:39.090146 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 317/625] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:31:39.090146 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5350240c-ab31-40e4-bb04-a8d15bf46bec None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:39.279413 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5350240c-ab31-40e4-bb04-a8d15bf46bec None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:39.312358 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd63503-3552-4392-8e0c-403a900d43cb None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:39.324909 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af91c87a-0496-4a16-ac15-1f50c276eadf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:39.325889 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af91c87a-0496-4a16-ac15-1f50c276eadf None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:39.326372 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af91c87a-0496-4a16-ac15-1f50c276eadf None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:39.326887 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af91c87a-0496-4a16-ac15-1f50c276eadf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:39.349764 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af91c87a-0496-4a16-ac15-1f50c276eadf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:39.355058 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af91c87a-0496-4a16-ac15-1f50c276eadf None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:39.382348 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-af91c87a-0496-4a16-ac15-1f50c276eadf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:39.382348 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:39.382348 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:39.382348 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:39.382348 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:39.382348 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:39.382348 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:39.382348 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:39.382348 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:39.382348 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:39.382348 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:39.382348 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:39.382348 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:39.382348 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:39.382348 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:39.382348 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:39.383414 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:39.383414 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:39.383414 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:39.383414 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:39.383414 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:39.383414 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:39.383414 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:39.383414 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:39.383414 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:39.383414 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:39.383414 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:39.383414 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:39.383414 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:39.383414 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:39.383414 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:39.383414 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:39.383414 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:39.384317 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:39.384317 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:39.384317 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:39.384317 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 318/626] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 296 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:39.385000 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a828592e-57c8-430a-a67b-689e8ff7b052 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:39.385244 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a828592e-57c8-430a-a67b-689e8ff7b052 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:39.385516 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a828592e-57c8-430a-a67b-689e8ff7b052 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:39.388055 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff3c75e-d6dd-4b4f-964a-c8bb07ba5395 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:39.388943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff3c75e-d6dd-4b4f-964a-c8bb07ba5395 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:39.389432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff3c75e-d6dd-4b4f-964a-c8bb07ba5395 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:39.389913 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff3c75e-d6dd-4b4f-964a-c8bb07ba5395 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:39.391513 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 319/627] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:39 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:39.392325 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a828592e-57c8-430a-a67b-689e8ff7b052 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:39.441123 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:39.442288 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:39.471384 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff3c75e-d6dd-4b4f-964a-c8bb07ba5395 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:39.473091 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff3c75e-d6dd-4b4f-964a-c8bb07ba5395 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:39.517969 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a828592e-57c8-430a-a67b-689e8ff7b052 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:39.563680 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f3b9e1-fff4-41a4-8ab4-e702eb00fc76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:39.564177 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f3b9e1-fff4-41a4-8ab4-e702eb00fc76 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:39.564458 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f3b9e1-fff4-41a4-8ab4-e702eb00fc76 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:39.564732 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f3b9e1-fff4-41a4-8ab4-e702eb00fc76 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:39.565616 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f3b9e1-fff4-41a4-8ab4-e702eb00fc76 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:39.565925 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f3b9e1-fff4-41a4-8ab4-e702eb00fc76 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:39.567907 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f3b9e1-fff4-41a4-8ab4-e702eb00fc76 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:39.585777 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 320/628] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:39.585777 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d141c92-c688-465c-98b2-55fac1a3fb17 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:39.585777 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d141c92-c688-465c-98b2-55fac1a3fb17 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:39.585777 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d141c92-c688-465c-98b2-55fac1a3fb17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:39.594953 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 309/629] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:31:39 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 523 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 09:31:39.595819 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff3c75e-d6dd-4b4f-964a-c8bb07ba5395 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:39.605419 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-1d141c92-c688-465c-98b2-55fac1a3fb17 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:39.607358 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-1d141c92-c688-465c-98b2-55fac1a3fb17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:39.679584 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:39.684731 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 321/630] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:31:39.685684 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d141c92-c688-465c-98b2-55fac1a3fb17 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:39.711083 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:39.713035 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff3c75e-d6dd-4b4f-964a-c8bb07ba5395 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:39.748240 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-105991e6-c96e-4031-a235-30ab3b020030 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:39.748875 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105991e6-c96e-4031-a235-30ab3b020030 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:39.749153 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105991e6-c96e-4031-a235-30ab3b020030 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:39.749426 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105991e6-c96e-4031-a235-30ab3b020030 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:39.760594 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-105991e6-c96e-4031-a235-30ab3b020030 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:31:39.765333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-105991e6-c96e-4031-a235-30ab3b020030 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 09:31:39.765333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:39.765333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:39.765333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:39.765333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:39.765333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:39.765333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:39.765333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:39.765333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:39.765333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:39.765333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:39.765333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:39.765333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:31:39.765333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:31:39.765333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:31:39.765333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:31:39.766277 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:31:39.766277 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:31:39.766277 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:31:39.766277 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:31:39.766277 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:39.766277 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:39.766277 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:39.766277 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:39.766277 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:39.766277 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:39.766277 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:39.766277 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:39.766277 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:39.766277 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:39.766277 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:39.766277 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:39.767181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:39.767181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:39.767181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:39.767181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:39.767181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:31:39.767181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:31:39.767181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:31:39.767181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:31:39.767181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:31:39.767181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:31:39.767181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 09:31:39.767181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:39.767181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 310/631] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Oct 28 09:31:39 2022] GET /identity/v3/roles/service => generated 85 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:39.767917 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-105991e6-c96e-4031-a235-30ab3b020030 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:39.802377 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d141c92-c688-465c-98b2-55fac1a3fb17 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:39.839044 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cef8b56-5428-40e3-9f85-047cdbfd27d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:39.840038 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cef8b56-5428-40e3-9f85-047cdbfd27d5 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:39.840549 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cef8b56-5428-40e3-9f85-047cdbfd27d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:39.841035 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cef8b56-5428-40e3-9f85-047cdbfd27d5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:39.848234 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cef8b56-5428-40e3-9f85-047cdbfd27d5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:39.849838 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cef8b56-5428-40e3-9f85-047cdbfd27d5 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:39.861421 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-2cef8b56-5428-40e3-9f85-047cdbfd27d5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:31:39.861421 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:39.861421 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:39.861421 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:39.861421 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:39.861421 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:39.861421 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:39.861421 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:39.861421 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:39.861421 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:39.861421 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:39.861421 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:39.861421 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 09:31:39.861421 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 09:31:39.861421 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 09:31:39.862359 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 09:31:39.862359 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:39.862359 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:39.862359 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:39.862359 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:39.862359 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:39.862359 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:39.862359 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:39.862359 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:39.862359 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:39.862359 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:39.862359 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:39.862359 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:39.862359 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:39.862359 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:39.862359 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:39.862359 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:39.863297 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 09:31:39.863297 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 09:31:39.863297 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 09:31:39.863297 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 09:31:39.863297 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 09:31:39.863297 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 09:31:39.863297 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:31:39.863297 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:39.867872 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 322/632] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:31:39 2022] GET /identity/v3/projects/service => generated 88 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:39.868974 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8e957e-3436-41cc-b183-b377d1734e12 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:39.869267 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8e957e-3436-41cc-b183-b377d1734e12 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:39.869545 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8e957e-3436-41cc-b183-b377d1734e12 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:39.870958 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 323/633] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:39.875260 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d8e957e-3436-41cc-b183-b377d1734e12 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:39.878193 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-105991e6-c96e-4031-a235-30ab3b020030 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:39.914861 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c2cc1e-eef2-4712-abe0-64a94fefea8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:39.915495 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c2cc1e-eef2-4712-abe0-64a94fefea8f None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:39.915773 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c2cc1e-eef2-4712-abe0-64a94fefea8f None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:39.916042 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c2cc1e-eef2-4712-abe0-64a94fefea8f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:39.919473 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:39.929608 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51c2cc1e-eef2-4712-abe0-64a94fefea8f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:39.931111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51c2cc1e-eef2-4712-abe0-64a94fefea8f None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:39.946848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-51c2cc1e-eef2-4712-abe0-64a94fefea8f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:39.946848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:39.946848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:39.946848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:39.946848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:39.946848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:39.946848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:39.946848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:39.946848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:39.946848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:39.946848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:39.946848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:39.946848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:39.946848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:39.946848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:39.946848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:39.948017 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:39.948017 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:39.948017 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:39.948017 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:39.948017 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:39.948017 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:39.948017 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:39.948017 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:39.948017 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:39.948017 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:39.948017 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:39.948017 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:39.948017 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:39.948017 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:39.948017 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:39.948017 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:39.948017 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:39.949092 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:39.949092 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:39.949092 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:39.952706 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 311/634] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:39.953574 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c2cc1e-eef2-4712-abe0-64a94fefea8f None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:39.968320 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:39.991792 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d8e957e-3436-41cc-b183-b377d1734e12 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:40.025675 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dc6d43b-f23c-492c-be7a-1874c71810c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:40.026341 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc6d43b-f23c-492c-be7a-1874c71810c2 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:40.026650 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc6d43b-f23c-492c-be7a-1874c71810c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:40.026921 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dc6d43b-f23c-492c-be7a-1874c71810c2 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:40.027574 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc6d43b-f23c-492c-be7a-1874c71810c2 None admin] RBAC: Adding query filter params (name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:40.027879 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc6d43b-f23c-492c-be7a-1874c71810c2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:40.029224 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dc6d43b-f23c-492c-be7a-1874c71810c2 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:40.037997 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 324/635] 10.222.0.25 () {56 vars in 1103 bytes} [Fri Oct 28 09:31:39 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:40.039771 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e7a94b-3278-4d65-bc28-f4bab837e605 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:40.040054 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e7a94b-3278-4d65-bc28-f4bab837e605 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:40.040325 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e7a94b-3278-4d65-bc28-f4bab837e605 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:40.053916 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c2cc1e-eef2-4712-abe0-64a94fefea8f None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:40.057788 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-30e7a94b-3278-4d65-bc28-f4bab837e605 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:40.059009 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-30e7a94b-3278-4d65-bc28-f4bab837e605 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:40.092855 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb0aa81-9dd7-4afe-b070-9d9c636c1652 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:40.093382 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb0aa81-9dd7-4afe-b070-9d9c636c1652 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:40.093664 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb0aa81-9dd7-4afe-b070-9d9c636c1652 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:40.093978 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb0aa81-9dd7-4afe-b070-9d9c636c1652 None admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:40.094655 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb0aa81-9dd7-4afe-b070-9d9c636c1652 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:40.094969 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb0aa81-9dd7-4afe-b070-9d9c636c1652 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:40.096399 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb0aa81-9dd7-4afe-b070-9d9c636c1652 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:40.108846 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 312/636] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Oct 28 09:31:39 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:40.109658 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb0aa81-9dd7-4afe-b070-9d9c636c1652 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:40.127814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 325/637] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:31:40.128670 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e7a94b-3278-4d65-bc28-f4bab837e605 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:40.169264 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:40.213939 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:40.228471 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb0aa81-9dd7-4afe-b070-9d9c636c1652 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:40.241362 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e7a94b-3278-4d65-bc28-f4bab837e605 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:40.265046 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6c58b6-d734-4c9d-80cf-69df40c6bae5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:40.265798 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6c58b6-d734-4c9d-80cf-69df40c6bae5 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:40.266185 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6c58b6-d734-4c9d-80cf-69df40c6bae5 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:40.266587 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6c58b6-d734-4c9d-80cf-69df40c6bae5 None admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:40.267333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb6c58b6-d734-4c9d-80cf-69df40c6bae5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:40.267774 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb6c58b6-d734-4c9d-80cf-69df40c6bae5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:40.269257 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb6c58b6-d734-4c9d-80cf-69df40c6bae5 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:40.276719 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e9f9371-80a1-46af-a9d7-08f75f972a35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:40.277380 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9f9371-80a1-46af-a9d7-08f75f972a35 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:40.277658 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9f9371-80a1-46af-a9d7-08f75f972a35 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:40.277994 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9f9371-80a1-46af-a9d7-08f75f972a35 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:40.286540 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 313/638] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:40.286540 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6c58b6-d734-4c9d-80cf-69df40c6bae5 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:40.294485 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e9f9371-80a1-46af-a9d7-08f75f972a35 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:31:40.299295 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-7e9f9371-80a1-46af-a9d7-08f75f972a35 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 09:31:40.299295 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:40.299295 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:40.299295 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:40.299295 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:40.299295 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:40.299295 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:40.299295 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:40.299295 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:40.299295 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:40.299295 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:40.299295 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:40.299295 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:31:40.299295 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:31:40.299295 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:31:40.299295 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:31:40.300261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:31:40.300261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:31:40.300261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:31:40.300261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:31:40.300261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:40.300261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:40.300261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:40.300261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:40.300261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:40.300261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:40.300261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:40.300261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:40.300261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:40.300261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:40.300261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:40.300261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:40.301094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:40.301094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:40.301094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:40.301094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:40.301094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:31:40.301094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:31:40.301094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:31:40.301094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:31:40.301094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:31:40.301094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:31:40.301094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 09:31:40.301094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:40.301094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 326/639] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Oct 28 09:31:40 2022] GET /identity/v3/roles/service => generated 85 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:40.301856 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e9f9371-80a1-46af-a9d7-08f75f972a35 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:40.397708 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6c58b6-d734-4c9d-80cf-69df40c6bae5 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:40.407190 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:40.411073 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e9f9371-80a1-46af-a9d7-08f75f972a35 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:40.432775 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5158675-dfc2-42f5-a94a-ff8f9002d10a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:40.433405 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5158675-dfc2-42f5-a94a-ff8f9002d10a None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:40.433683 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5158675-dfc2-42f5-a94a-ff8f9002d10a None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:40.433963 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5158675-dfc2-42f5-a94a-ff8f9002d10a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:40.445420 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5158675-dfc2-42f5-a94a-ff8f9002d10a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:40.446994 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc4b8d2-ee6a-43cb-a918-3da67761e9f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:40.447182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5158675-dfc2-42f5-a94a-ff8f9002d10a None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:40.447662 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc4b8d2-ee6a-43cb-a918-3da67761e9f5 None admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:40.447943 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc4b8d2-ee6a-43cb-a918-3da67761e9f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:40.448218 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc4b8d2-ee6a-43cb-a918-3da67761e9f5 None admin] PATH_INFO: `/v3/projects/455d1c826b974bc38497d18a516867c7/users/9c6cffe0d56e45c4b8fb54fbe5df6094/roles/e5d8d090857a4a018d0ec3c79ece6472` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:40.450880 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:40.463054 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-f5158675-dfc2-42f5-a94a-ff8f9002d10a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:40.463054 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:40.463054 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:40.463054 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:40.463054 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:40.463054 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:40.463054 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:40.463054 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:40.463054 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:40.463054 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:40.463054 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:40.463054 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:40.463054 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:40.463054 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:40.463054 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:40.463054 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:40.464049 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:40.464049 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:40.464049 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:40.464049 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:40.464049 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:40.464049 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:40.464049 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:40.464049 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:40.464049 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:40.464049 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:40.464049 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:40.464049 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:40.464049 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:40.464049 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:40.464049 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:40.464049 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:40.464049 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:40.464930 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:40.464930 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:40.464930 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:40.465134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc4b8d2-ee6a-43cb-a918-3da67761e9f5 None admin] RBAC: Authorizing `identity:create_grant(project_id=455d1c826b974bc38497d18a516867c7, user_id=9c6cffe0d56e45c4b8fb54fbe5df6094, role_id=e5d8d090857a4a018d0ec3c79ece6472)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:40.466674 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 314/640] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:40.466784 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc4b8d2-ee6a-43cb-a918-3da67761e9f5 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:40.467377 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5158675-dfc2-42f5-a94a-ff8f9002d10a None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:40.479924 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 327/641] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Oct 28 09:31:40 2022] PUT /identity/v3/projects/455d1c826b974bc38497d18a516867c7/users/9c6cffe0d56e45c4b8fb54fbe5df6094/roles/e5d8d090857a4a018d0ec3c79ece6472 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:31:40.480770 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc4b8d2-ee6a-43cb-a918-3da67761e9f5 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:40.582153 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5158675-dfc2-42f5-a94a-ff8f9002d10a None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:40.616047 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc4b8d2-ee6a-43cb-a918-3da67761e9f5 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:40.621105 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f18ba0-30cb-4bf2-b165-82b579c9ee00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:40.621809 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f18ba0-30cb-4bf2-b165-82b579c9ee00 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:40.622094 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f18ba0-30cb-4bf2-b165-82b579c9ee00 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:40.622370 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f18ba0-30cb-4bf2-b165-82b579c9ee00 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:40.630107 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29f18ba0-30cb-4bf2-b165-82b579c9ee00 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:40.631638 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29f18ba0-30cb-4bf2-b165-82b579c9ee00 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:40.643936 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-29f18ba0-30cb-4bf2-b165-82b579c9ee00 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:31:40.643936 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:40.643936 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:40.643936 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:40.643936 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:40.643936 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:40.643936 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:40.643936 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:40.643936 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:40.643936 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:40.643936 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:40.643936 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:40.643936 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 09:31:40.643936 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 09:31:40.643936 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 09:31:40.644970 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 09:31:40.644970 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:40.644970 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:40.644970 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:40.644970 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:40.644970 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:40.644970 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:40.644970 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:40.644970 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:40.644970 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:40.644970 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:40.644970 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:40.644970 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:40.644970 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:40.644970 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:40.644970 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:40.644970 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:40.645858 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 09:31:40.645858 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 09:31:40.645858 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 09:31:40.645858 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 09:31:40.645858 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 09:31:40.645858 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 09:31:40.645858 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:31:40.645858 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:40.645858 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 315/642] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:31:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:40.645858 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f18ba0-30cb-4bf2-b165-82b579c9ee00 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:40.666006 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1792626-f9b6-4b27-85b4-6c085dbe26c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:40.666797 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1792626-f9b6-4b27-85b4-6c085dbe26c7 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:40.666921 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1792626-f9b6-4b27-85b4-6c085dbe26c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:40.667108 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1792626-f9b6-4b27-85b4-6c085dbe26c7 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:40.667788 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1792626-f9b6-4b27-85b4-6c085dbe26c7 None admin] RBAC: Adding query filter params (name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:40.668093 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1792626-f9b6-4b27-85b4-6c085dbe26c7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:40.669477 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1792626-f9b6-4b27-85b4-6c085dbe26c7 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:40.679126 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:40.683467 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 328/643] 10.222.0.25 () {56 vars in 1103 bytes} [Fri Oct 28 09:31:40 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:40.684628 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1792626-f9b6-4b27-85b4-6c085dbe26c7 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:40.737439 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:40.775626 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f18ba0-30cb-4bf2-b165-82b579c9ee00 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:40.804346 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1792626-f9b6-4b27-85b4-6c085dbe26c7 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:40.810996 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a6c4fdf-cc56-496c-84c0-3fadcf45c8ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:40.811862 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6c4fdf-cc56-496c-84c0-3fadcf45c8ac None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:40.812146 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6c4fdf-cc56-496c-84c0-3fadcf45c8ac None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:40.812419 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6c4fdf-cc56-496c-84c0-3fadcf45c8ac None admin] PATH_INFO: `/v3/roles/service` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:40.832276 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a6c4fdf-cc56-496c-84c0-3fadcf45c8ac None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:31:40.838158 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-3a6c4fdf-cc56-496c-84c0-3fadcf45c8ac None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 09:31:40.838158 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:40.838158 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:40.838158 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:40.838158 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:40.838158 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:40.838158 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:40.838158 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:40.838158 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:40.838158 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:40.838158 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:40.838158 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:40.838158 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:31:40.838158 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:31:40.838158 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:31:40.838158 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:31:40.839109 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:31:40.839109 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:31:40.839109 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:31:40.839109 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:31:40.839109 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:40.839109 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:40.839109 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:40.839109 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:40.839109 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:40.839109 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:40.839109 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:40.839109 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:40.839109 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:40.839109 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:40.839109 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:40.839109 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:40.839998 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:40.839998 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:40.839998 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:40.839998 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:40.839998 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:31:40.839998 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:31:40.839998 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:31:40.839998 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:31:40.839998 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:31:40.839998 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:31:40.839998 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 09:31:40.839998 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:40.839998 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 316/644] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Oct 28 09:31:40 2022] GET /identity/v3/roles/service => generated 85 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:40.840817 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a6c4fdf-cc56-496c-84c0-3fadcf45c8ac None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:40.852057 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ab8050-0140-4aa0-bf8e-78c7588f5576 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:40.852998 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ab8050-0140-4aa0-bf8e-78c7588f5576 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:40.853287 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ab8050-0140-4aa0-bf8e-78c7588f5576 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:40.853727 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ab8050-0140-4aa0-bf8e-78c7588f5576 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:40.854376 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ab8050-0140-4aa0-bf8e-78c7588f5576 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:40.854726 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ab8050-0140-4aa0-bf8e-78c7588f5576 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:40.856086 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ab8050-0140-4aa0-bf8e-78c7588f5576 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:40.873270 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 329/645] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:40.873270 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ab8050-0140-4aa0-bf8e-78c7588f5576 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:40.936436 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:40.957165 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a6c4fdf-cc56-496c-84c0-3fadcf45c8ac None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:40.981361 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ab8050-0140-4aa0-bf8e-78c7588f5576 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:40.989742 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:40.996955 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f939f8cd-8422-4b32-87a0-62007a245d21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:40.997650 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f939f8cd-8422-4b32-87a0-62007a245d21 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:40.997996 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f939f8cd-8422-4b32-87a0-62007a245d21 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:40.998332 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f939f8cd-8422-4b32-87a0-62007a245d21 None admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:40.999114 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f939f8cd-8422-4b32-87a0-62007a245d21 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:40.999484 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f939f8cd-8422-4b32-87a0-62007a245d21 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:41.000980 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f939f8cd-8422-4b32-87a0-62007a245d21 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:41.019348 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 317/646] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Oct 28 09:31:40 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:41.019348 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f939f8cd-8422-4b32-87a0-62007a245d21 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:41.026643 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93cb6a7-19d3-4305-8334-9187fa5a9f12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:41.027285 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93cb6a7-19d3-4305-8334-9187fa5a9f12 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:41.027568 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93cb6a7-19d3-4305-8334-9187fa5a9f12 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:41.027840 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93cb6a7-19d3-4305-8334-9187fa5a9f12 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:41.043841 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f93cb6a7-19d3-4305-8334-9187fa5a9f12 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:41.045288 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f93cb6a7-19d3-4305-8334-9187fa5a9f12 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:41.061061 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-f93cb6a7-19d3-4305-8334-9187fa5a9f12 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:41.061061 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:41.061061 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:41.061061 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:41.061061 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:41.061061 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:41.061061 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:41.061061 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:41.061061 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:41.061061 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:41.061061 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:41.061061 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:41.061061 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:41.061061 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:41.061061 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:41.061061 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:41.062252 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:41.062252 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:41.062252 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:41.062252 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:41.062252 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:41.062252 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:41.062252 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:41.062252 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:41.062252 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:41.062252 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:41.062252 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:41.062252 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:41.062252 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:41.062252 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:41.062252 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:41.062252 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:41.062252 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:41.063220 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:41.063220 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:41.063220 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:41.063220 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 330/647] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:41.063455 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93cb6a7-19d3-4305-8334-9187fa5a9f12 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:41.149230 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f939f8cd-8422-4b32-87a0-62007a245d21 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:41.179711 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93cb6a7-19d3-4305-8334-9187fa5a9f12 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:41.202256 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f47797c7-439f-44a2-91fe-e16175874b5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:41.203522 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47797c7-439f-44a2-91fe-e16175874b5d None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:41.203964 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47797c7-439f-44a2-91fe-e16175874b5d None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:41.204396 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47797c7-439f-44a2-91fe-e16175874b5d None admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:41.205196 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f47797c7-439f-44a2-91fe-e16175874b5d None admin] RBAC: Adding query filter params (name=service) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:41.205639 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f47797c7-439f-44a2-91fe-e16175874b5d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:41.207204 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f47797c7-439f-44a2-91fe-e16175874b5d None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:41.219446 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 318/648] 10.222.0.25 () {56 vars in 1103 bytes} [Fri Oct 28 09:31:41 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:41.220469 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f47797c7-439f-44a2-91fe-e16175874b5d None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:41.224304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:41.232336 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade12207-dcd1-426e-a066-9ddb004e49ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:41.233043 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade12207-dcd1-426e-a066-9ddb004e49ad None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:41.233327 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade12207-dcd1-426e-a066-9ddb004e49ad None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:41.233598 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade12207-dcd1-426e-a066-9ddb004e49ad None admin] PATH_INFO: `/v3/users/glance` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:41.241743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ade12207-dcd1-426e-a066-9ddb004e49ad None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:41.243613 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ade12207-dcd1-426e-a066-9ddb004e49ad None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:41.257695 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-ade12207-dcd1-426e-a066-9ddb004e49ad None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Oct 28 09:31:41.257695 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:41.257695 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:41.257695 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:41.257695 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:41.257695 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:41.257695 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:41.257695 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:41.257695 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:41.257695 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:41.257695 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:41.257695 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:41.257695 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 09:31:41.257695 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 09:31:41.257695 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 09:31:41.257695 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 09:31:41.258734 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:41.258734 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:41.258734 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 09:31:41.258734 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:31:41.258734 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 09:31:41.258734 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:31:41.258734 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:41.258734 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:41.258734 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:41.258734 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:41.258734 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:41.258734 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:41.258734 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:41.258734 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:41.258734 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:41.258734 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:41.258734 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:41.258734 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:41.259726 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:41.259726 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:41.259726 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 09:31:41.259726 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 09:31:41.259726 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 09:31:41.259726 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 09:31:41.259726 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 09:31:41.259726 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 09:31:41.259726 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Oct 28 09:31:41.259726 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:41.259726 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 331/649] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:31:41 2022] GET /identity/v3/users/glance => generated 84 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:41.259726 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade12207-dcd1-426e-a066-9ddb004e49ad None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:41.280006 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:41.331459 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f47797c7-439f-44a2-91fe-e16175874b5d None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:41.368826 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918ef03c-a1f4-49dc-8661-824a3c839978 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:41.369425 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918ef03c-a1f4-49dc-8661-824a3c839978 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:41.369707 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918ef03c-a1f4-49dc-8661-824a3c839978 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:41.369984 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-918ef03c-a1f4-49dc-8661-824a3c839978 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:41.370752 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-918ef03c-a1f4-49dc-8661-824a3c839978 None admin] RBAC: Adding query filter params (role.id=e5d8d090857a4a018d0ec3c79ece6472, scope.project.id=455d1c826b974bc38497d18a516867c7, user.id=688ceb201d304c2da5373c3cf36c2d79) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:41.371066 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-918ef03c-a1f4-49dc-8661-824a3c839978 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:41.372535 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-918ef03c-a1f4-49dc-8661-824a3c839978 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:41.383079 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 319/650] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:31:41 2022] GET /identity/v3/role_assignments?user.id=688ceb201d304c2da5373c3cf36c2d79&scope.project.id=455d1c826b974bc38497d18a516867c7&role.id=e5d8d090857a4a018d0ec3c79ece6472 => generated 608 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:41.383079 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918ef03c-a1f4-49dc-8661-824a3c839978 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:41.386804 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade12207-dcd1-426e-a066-9ddb004e49ad None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:41.433517 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6627b249-7196-46b2-bf73-dc16733ecc2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:41.434655 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6627b249-7196-46b2-bf73-dc16733ecc2d None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:41.435151 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6627b249-7196-46b2-bf73-dc16733ecc2d None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:41.435500 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6627b249-7196-46b2-bf73-dc16733ecc2d None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:41.436239 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6627b249-7196-46b2-bf73-dc16733ecc2d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:41.436619 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6627b249-7196-46b2-bf73-dc16733ecc2d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:41.438180 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6627b249-7196-46b2-bf73-dc16733ecc2d None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:41.460658 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 332/651] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:41.460658 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6627b249-7196-46b2-bf73-dc16733ecc2d None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:41.482878 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:41.527830 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-918ef03c-a1f4-49dc-8661-824a3c839978 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:41.563471 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:41.583792 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cbe1947-7033-4ece-b3e2-4e3db4dbbb97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:41.585585 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbe1947-7033-4ece-b3e2-4e3db4dbbb97 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:41.585585 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbe1947-7033-4ece-b3e2-4e3db4dbbb97 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:41.585585 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbe1947-7033-4ece-b3e2-4e3db4dbbb97 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:41.597487 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cbe1947-7033-4ece-b3e2-4e3db4dbbb97 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:41.599007 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cbe1947-7033-4ece-b3e2-4e3db4dbbb97 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:41.608130 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6627b249-7196-46b2-bf73-dc16733ecc2d None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:41.618263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-1cbe1947-7033-4ece-b3e2-4e3db4dbbb97 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:41.618263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:41.618263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:41.618263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:41.618263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:41.618263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:41.618263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:41.618263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:41.618263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:41.618263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:41.618263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:41.618263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:41.618263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:41.618263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:41.618263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:41.618263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:41.619304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:41.619304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:41.619304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:41.619304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:41.619304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:41.619304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:41.619304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:41.619304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:41.619304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:41.619304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:41.619304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:41.619304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:41.619304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:41.619304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:41.619304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:41.619304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:41.619304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:41.620620 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:41.620620 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:41.620620 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:41.620620 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 320/652] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:41.620620 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cbe1947-7033-4ece-b3e2-4e3db4dbbb97 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:41.657134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc268bff-30fa-470e-85c2-1c02577af23a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:41.658016 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc268bff-30fa-470e-85c2-1c02577af23a None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:41.658303 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc268bff-30fa-470e-85c2-1c02577af23a None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:41.658612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc268bff-30fa-470e-85c2-1c02577af23a None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:41.659462 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc268bff-30fa-470e-85c2-1c02577af23a None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:41.659775 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc268bff-30fa-470e-85c2-1c02577af23a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:41.661195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc268bff-30fa-470e-85c2-1c02577af23a None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:41.690709 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 333/653] 10.222.0.25 () {56 vars in 1137 bytes} [Fri Oct 28 09:31:41 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:41.690709 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc268bff-30fa-470e-85c2-1c02577af23a None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:41.764776 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cbe1947-7033-4ece-b3e2-4e3db4dbbb97 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:41.784989 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:41.805440 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-991b48ea-cf23-4d6b-9ce9-2b9108a8e9e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:41.806182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991b48ea-cf23-4d6b-9ce9-2b9108a8e9e9 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:41.806579 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991b48ea-cf23-4d6b-9ce9-2b9108a8e9e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:41.806929 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991b48ea-cf23-4d6b-9ce9-2b9108a8e9e9 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:41.818806 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-991b48ea-cf23-4d6b-9ce9-2b9108a8e9e9 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:41.820790 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc268bff-30fa-470e-85c2-1c02577af23a None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:41.821193 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-991b48ea-cf23-4d6b-9ce9-2b9108a8e9e9 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:41.836319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-991b48ea-cf23-4d6b-9ce9-2b9108a8e9e9 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 28 09:31:41.836319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:41.836319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:41.836319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:41.836319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:41.836319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:41.836319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:41.836319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:41.836319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:41.836319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:41.836319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:41.836319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:41.836319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 09:31:41.836319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 09:31:41.836319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 09:31:41.836319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 09:31:41.837323 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:41.837323 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:41.837323 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 09:31:41.837323 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:31:41.837323 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 09:31:41.837323 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:31:41.837323 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:41.837323 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:41.837323 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:41.837323 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:41.837323 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:41.837323 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:41.837323 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:41.837323 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:41.837323 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:41.837323 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:41.837323 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:41.837323 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:41.838276 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:41.838276 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:41.838276 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 09:31:41.838276 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 09:31:41.838276 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 09:31:41.838276 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 09:31:41.838276 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 09:31:41.838276 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 09:31:41.838276 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 28 09:31:41.838276 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:41.840167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 321/654] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:31:41 2022] GET /identity/v3/users/cinder => generated 84 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:41.840167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-991b48ea-cf23-4d6b-9ce9-2b9108a8e9e9 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:41.863736 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f47110-5f21-4321-a9a6-638505caab69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:41.864354 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f47110-5f21-4321-a9a6-638505caab69 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:41.864633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f47110-5f21-4321-a9a6-638505caab69 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:41.864908 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f47110-5f21-4321-a9a6-638505caab69 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:41.865556 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f47110-5f21-4321-a9a6-638505caab69 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:41.865857 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f47110-5f21-4321-a9a6-638505caab69 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:41.867460 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f47110-5f21-4321-a9a6-638505caab69 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:41.870028 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:41.882535 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 334/655] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:41.883767 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f47110-5f21-4321-a9a6-638505caab69 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:41.975048 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-991b48ea-cf23-4d6b-9ce9-2b9108a8e9e9 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:41.994912 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f47110-5f21-4321-a9a6-638505caab69 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:42.027066 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18edb4e-b64a-4e8b-a1d5-16e1dd28d80b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:42.027984 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18edb4e-b64a-4e8b-a1d5-16e1dd28d80b None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:42.028340 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18edb4e-b64a-4e8b-a1d5-16e1dd28d80b None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:42.028688 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18edb4e-b64a-4e8b-a1d5-16e1dd28d80b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:42.039114 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ab6a27-48ce-4e95-9b6e-9c6ed96e1f12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:42.040333 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ab6a27-48ce-4e95-9b6e-9c6ed96e1f12 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:42.040807 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ab6a27-48ce-4e95-9b6e-9c6ed96e1f12 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:42.041276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ab6a27-48ce-4e95-9b6e-9c6ed96e1f12 None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:42.042178 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ab6a27-48ce-4e95-9b6e-9c6ed96e1f12 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:42.042697 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ab6a27-48ce-4e95-9b6e-9c6ed96e1f12 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:42.044412 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ab6a27-48ce-4e95-9b6e-9c6ed96e1f12 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:42.068006 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 335/656] 10.222.0.25 () {56 vars in 1137 bytes} [Fri Oct 28 09:31:41 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:42.069242 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ab6a27-48ce-4e95-9b6e-9c6ed96e1f12 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:42.092054 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:42.129473 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f18edb4e-b64a-4e8b-a1d5-16e1dd28d80b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:42.131616 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f18edb4e-b64a-4e8b-a1d5-16e1dd28d80b None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:42.133919 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:42.185897 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ab6a27-48ce-4e95-9b6e-9c6ed96e1f12 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:42.224252 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc2eca7-b9bf-4c6b-b5df-37bac2e78e91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:42.225250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc2eca7-b9bf-4c6b-b5df-37bac2e78e91 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:42.225800 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc2eca7-b9bf-4c6b-b5df-37bac2e78e91 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:42.226287 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fc2eca7-b9bf-4c6b-b5df-37bac2e78e91 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:42.234955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fc2eca7-b9bf-4c6b-b5df-37bac2e78e91 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:42.236792 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fc2eca7-b9bf-4c6b-b5df-37bac2e78e91 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:42.251248 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-2fc2eca7-b9bf-4c6b-b5df-37bac2e78e91 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Oct 28 09:31:42.251248 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:42.251248 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:42.251248 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:42.251248 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:42.251248 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:42.251248 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:42.251248 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:42.251248 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:42.251248 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:42.251248 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:42.251248 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:42.251248 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 09:31:42.251248 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 09:31:42.251248 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 09:31:42.251248 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 09:31:42.252240 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:42.252240 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:42.252240 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 09:31:42.252240 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:31:42.252240 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 09:31:42.252240 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:31:42.252240 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:42.252240 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:42.252240 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:42.252240 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:42.252240 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:42.252240 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:42.252240 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:42.252240 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:42.252240 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:42.252240 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:42.252240 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:42.252240 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:42.253160 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:42.253160 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:42.253160 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 09:31:42.253160 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 09:31:42.253160 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 09:31:42.253160 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 09:31:42.253160 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 09:31:42.253160 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 09:31:42.253160 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Oct 28 09:31:42.253160 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:42.255917 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 336/657] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Oct 28 09:31:42 2022] GET /identity/v3/users/neutron => generated 85 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:42.257685 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc2eca7-b9bf-4c6b-b5df-37bac2e78e91 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:42.270512 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 322/658] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:31:41 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 427 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 09:31:42.273032 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18edb4e-b64a-4e8b-a1d5-16e1dd28d80b None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:42.367559 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:42.397234 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18edb4e-b64a-4e8b-a1d5-16e1dd28d80b None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:42.400261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fc2eca7-b9bf-4c6b-b5df-37bac2e78e91 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:42.416867 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:42.444197 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6023865b-b4a4-4e72-9ba3-2ea97a8fddb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:42.445461 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6023865b-b4a4-4e72-9ba3-2ea97a8fddb5 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:42.445963 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6023865b-b4a4-4e72-9ba3-2ea97a8fddb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:42.446377 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6023865b-b4a4-4e72-9ba3-2ea97a8fddb5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:42.447725 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aee58bb-0064-4f5e-967e-76a543d60900 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:42.448435 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aee58bb-0064-4f5e-967e-76a543d60900 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:42.448808 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aee58bb-0064-4f5e-967e-76a543d60900 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:42.449198 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aee58bb-0064-4f5e-967e-76a543d60900 None admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:42.450311 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aee58bb-0064-4f5e-967e-76a543d60900 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:42.450737 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aee58bb-0064-4f5e-967e-76a543d60900 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:42.453244 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aee58bb-0064-4f5e-967e-76a543d60900 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:42.477761 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 323/659] 10.222.0.25 () {56 vars in 1139 bytes} [Fri Oct 28 09:31:42 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:42.479358 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aee58bb-0064-4f5e-967e-76a543d60900 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:42.523635 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6023865b-b4a4-4e72-9ba3-2ea97a8fddb5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:42.525416 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6023865b-b4a4-4e72-9ba3-2ea97a8fddb5 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:42.617987 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aee58bb-0064-4f5e-967e-76a543d60900 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:42.653681 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 337/660] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:31:42 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 392 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 09:31:42.655164 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6023865b-b4a4-4e72-9ba3-2ea97a8fddb5 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:42.662781 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22756b42-4033-45aa-aff0-909c9a77b7cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:42.664069 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22756b42-4033-45aa-aff0-909c9a77b7cc None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:42.664443 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22756b42-4033-45aa-aff0-909c9a77b7cc None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:42.664814 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22756b42-4033-45aa-aff0-909c9a77b7cc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:42.668405 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:42.676615 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:42.682286 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22756b42-4033-45aa-aff0-909c9a77b7cc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:42.684376 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22756b42-4033-45aa-aff0-909c9a77b7cc None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:42.701968 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-22756b42-4033-45aa-aff0-909c9a77b7cc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:42.701968 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:42.701968 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:42.701968 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:42.701968 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:42.701968 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:42.701968 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:42.701968 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:42.701968 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:42.701968 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:42.701968 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:42.701968 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:42.701968 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:42.701968 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:42.701968 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:42.701968 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:42.703190 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:42.703190 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:42.703190 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:42.703190 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:42.703190 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:42.703190 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:42.703190 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:42.703190 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:42.703190 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:42.703190 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:42.703190 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:42.703190 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:42.703190 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:42.703190 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:42.703190 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:42.703190 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:42.703190 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:42.704265 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:42.704265 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:42.704265 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:42.706138 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 324/661] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:42.707034 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22756b42-4033-45aa-aff0-909c9a77b7cc None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:42.778791 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6023865b-b4a4-4e72-9ba3-2ea97a8fddb5 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:42.816771 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db98705-d30d-41d3-9e3a-14c213ef2867 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:42.817379 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db98705-d30d-41d3-9e3a-14c213ef2867 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:42.817659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db98705-d30d-41d3-9e3a-14c213ef2867 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:42.817932 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db98705-d30d-41d3-9e3a-14c213ef2867 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:42.861753 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22756b42-4033-45aa-aff0-909c9a77b7cc None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:42.892648 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5db98705-d30d-41d3-9e3a-14c213ef2867 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:42.894014 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5db98705-d30d-41d3-9e3a-14c213ef2867 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:42.909867 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:42.918090 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b174fd30-9a34-4344-a241-33e600ce333d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:42.919132 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b174fd30-9a34-4344-a241-33e600ce333d None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:42.919613 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b174fd30-9a34-4344-a241-33e600ce333d None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:42.920101 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b174fd30-9a34-4344-a241-33e600ce333d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:42.937571 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b174fd30-9a34-4344-a241-33e600ce333d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:42.940074 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b174fd30-9a34-4344-a241-33e600ce333d None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:42.966966 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-b174fd30-9a34-4344-a241-33e600ce333d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:42.966966 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:42.966966 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:42.966966 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:42.966966 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:42.966966 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:42.966966 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:42.966966 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:42.966966 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:42.966966 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:42.966966 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:42.966966 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:42.966966 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:42.966966 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:42.966966 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:42.966966 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:42.967946 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:42.967946 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:42.967946 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:42.967946 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:42.967946 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:42.967946 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:42.967946 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:42.967946 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:42.967946 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:42.967946 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:42.967946 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:42.967946 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:42.967946 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:42.967946 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:42.967946 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:42.967946 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:42.967946 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:42.968828 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:42.968828 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:42.968828 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:42.968997 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 325/662] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 262 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:42.970902 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa069a13-f66b-4aa9-ba6b-b263b8b6a635 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:42.971400 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa069a13-f66b-4aa9-ba6b-b263b8b6a635 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:42.971877 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa069a13-f66b-4aa9-ba6b-b263b8b6a635 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:42.974135 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 326/663] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:42 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:42.975398 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa069a13-f66b-4aa9-ba6b-b263b8b6a635 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:43.008267 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 338/664] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:31:42 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 354 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 09:31:43.009122 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db98705-d30d-41d3-9e3a-14c213ef2867 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:43.028331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:43.092088 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa069a13-f66b-4aa9-ba6b-b263b8b6a635 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:43.119949 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db98705-d30d-41d3-9e3a-14c213ef2867 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:43.129240 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8683c12b-153e-43e7-b32d-8db20473d732 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:43.130244 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8683c12b-153e-43e7-b32d-8db20473d732 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:43.130566 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8683c12b-153e-43e7-b32d-8db20473d732 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:43.130847 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8683c12b-153e-43e7-b32d-8db20473d732 None admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:43.131831 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8683c12b-153e-43e7-b32d-8db20473d732 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:43.132159 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8683c12b-153e-43e7-b32d-8db20473d732 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:43.133649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8683c12b-153e-43e7-b32d-8db20473d732 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:43.135318 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:43.150800 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 327/665] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:43.150800 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9319dd0-6645-4c18-8cd0-dfd9e4672223 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:43.150800 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9319dd0-6645-4c18-8cd0-dfd9e4672223 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:43.150800 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9319dd0-6645-4c18-8cd0-dfd9e4672223 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:43.164021 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-783c3744-8282-4a49-9bc8-d1d5623bae52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:43.164716 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783c3744-8282-4a49-9bc8-d1d5623bae52 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:43.165065 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783c3744-8282-4a49-9bc8-d1d5623bae52 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:43.165426 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783c3744-8282-4a49-9bc8-d1d5623bae52 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:43.166135 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-783c3744-8282-4a49-9bc8-d1d5623bae52 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:43.166531 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-783c3744-8282-4a49-9bc8-d1d5623bae52 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:43.166987 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-c9319dd0-6645-4c18-8cd0-dfd9e4672223 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:43.168013 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-783c3744-8282-4a49-9bc8-d1d5623bae52 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:43.168182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-c9319dd0-6645-4c18-8cd0-dfd9e4672223 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:43.185950 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 339/666] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:43.185950 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-783c3744-8282-4a49-9bc8-d1d5623bae52 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:43.246063 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 328/667] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:31:43.247406 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9319dd0-6645-4c18-8cd0-dfd9e4672223 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:43.297060 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:43.297611 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-783c3744-8282-4a49-9bc8-d1d5623bae52 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:43.333744 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1392b481-cc17-41a5-bda9-cd4251ff5b78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:43.334429 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1392b481-cc17-41a5-bda9-cd4251ff5b78 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:43.334715 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1392b481-cc17-41a5-bda9-cd4251ff5b78 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:43.335049 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1392b481-cc17-41a5-bda9-cd4251ff5b78 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:43.346690 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1392b481-cc17-41a5-bda9-cd4251ff5b78 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:43.348228 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1392b481-cc17-41a5-bda9-cd4251ff5b78 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:43.364219 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-1392b481-cc17-41a5-bda9-cd4251ff5b78 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:43.364219 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:43.364219 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:43.364219 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:43.364219 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:43.364219 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:43.364219 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:43.364219 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:43.364219 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:43.364219 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:43.364219 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:43.364219 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:43.364219 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:43.364219 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:43.364219 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:43.364219 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:43.365250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:43.365250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:43.365250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:43.365250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:43.365250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:43.365250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:43.365250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:43.365250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:43.365250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:43.365250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:43.365250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:43.365250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:43.365250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:43.365250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:43.365250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:43.365250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:43.365250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:43.366213 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:43.366213 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:43.366213 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:43.370004 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 340/668] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:43.370922 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1392b481-cc17-41a5-bda9-cd4251ff5b78 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:43.382456 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9319dd0-6645-4c18-8cd0-dfd9e4672223 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:43.405861 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:43.429585 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76522748-183f-41ca-b043-6ee3394b8c02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:43.430569 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76522748-183f-41ca-b043-6ee3394b8c02 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:43.430853 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76522748-183f-41ca-b043-6ee3394b8c02 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:43.431127 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76522748-183f-41ca-b043-6ee3394b8c02 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:43.438684 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76522748-183f-41ca-b043-6ee3394b8c02 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:43.440323 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76522748-183f-41ca-b043-6ee3394b8c02 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:43.453269 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-76522748-183f-41ca-b043-6ee3394b8c02 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:31:43.453269 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:43.453269 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:43.453269 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:43.453269 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:43.453269 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:43.453269 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:43.453269 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:43.453269 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:43.453269 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:43.453269 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:43.453269 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:43.453269 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 09:31:43.453269 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 09:31:43.453269 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 09:31:43.454225 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 09:31:43.454225 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:43.454225 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:43.454225 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:43.454225 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:43.454225 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:43.454225 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:43.454225 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:43.454225 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:43.454225 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:43.454225 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:43.454225 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:43.454225 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:43.454225 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:43.454225 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:43.454225 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:43.454225 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:43.455168 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 09:31:43.455168 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 09:31:43.455168 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 09:31:43.455168 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 09:31:43.455168 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 09:31:43.455168 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 09:31:43.455168 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:31:43.455168 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:43.455168 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 329/669] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:31:43 2022] GET /identity/v3/projects/service => generated 88 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:43.455659 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76522748-183f-41ca-b043-6ee3394b8c02 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:43.484268 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1392b481-cc17-41a5-bda9-cd4251ff5b78 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:43.520418 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffb6045-3793-4889-a219-ebab86ff9a0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:43.521075 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffb6045-3793-4889-a219-ebab86ff9a0b None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:43.521352 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffb6045-3793-4889-a219-ebab86ff9a0b None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:43.521624 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffb6045-3793-4889-a219-ebab86ff9a0b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:43.528829 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ffb6045-3793-4889-a219-ebab86ff9a0b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:43.530379 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ffb6045-3793-4889-a219-ebab86ff9a0b None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:43.543488 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-2ffb6045-3793-4889-a219-ebab86ff9a0b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:31:43.543488 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:43.543488 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:43.543488 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:43.543488 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:43.543488 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:43.543488 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:43.543488 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:43.543488 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:43.543488 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:43.543488 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:43.543488 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:43.543488 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 09:31:43.543488 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 09:31:43.543488 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 09:31:43.545518 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 09:31:43.545518 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:43.545518 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:43.545518 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:43.545518 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:43.545518 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:43.545518 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:43.545518 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:43.545518 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:43.545518 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:43.545518 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:43.545518 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:43.545518 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:43.545518 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:43.545518 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:43.545518 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:43.545518 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:43.547737 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 09:31:43.547737 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 09:31:43.547737 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 09:31:43.547737 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 09:31:43.547737 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 09:31:43.547737 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 09:31:43.547737 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:31:43.547737 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:43.547737 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 341/670] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:31:43 2022] GET /identity/v3/projects/service => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:43.547737 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffb6045-3793-4889-a219-ebab86ff9a0b None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:43.563534 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:43.577064 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76522748-183f-41ca-b043-6ee3394b8c02 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:43.613456 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1777ba82-2b66-46a9-b9e1-98b08047cbda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:43.614095 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1777ba82-2b66-46a9-b9e1-98b08047cbda None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:43.614375 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1777ba82-2b66-46a9-b9e1-98b08047cbda None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:43.614682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1777ba82-2b66-46a9-b9e1-98b08047cbda None admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:43.615379 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1777ba82-2b66-46a9-b9e1-98b08047cbda None admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:43.615684 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1777ba82-2b66-46a9-b9e1-98b08047cbda None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:43.617687 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1777ba82-2b66-46a9-b9e1-98b08047cbda None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:43.632634 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 330/671] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:43.633536 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1777ba82-2b66-46a9-b9e1-98b08047cbda None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:43.646540 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:43.664327 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffb6045-3793-4889-a219-ebab86ff9a0b None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:43.699594 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe6b18df-a7a6-484f-ae79-f9274db72eec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:43.700319 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6b18df-a7a6-484f-ae79-f9274db72eec None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:43.700598 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6b18df-a7a6-484f-ae79-f9274db72eec None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:43.700866 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6b18df-a7a6-484f-ae79-f9274db72eec None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:43.701550 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe6b18df-a7a6-484f-ae79-f9274db72eec None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:43.701853 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe6b18df-a7a6-484f-ae79-f9274db72eec None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:43.703589 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe6b18df-a7a6-484f-ae79-f9274db72eec None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:43.717680 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 342/672] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Oct 28 09:31:43 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:43.718331 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe6b18df-a7a6-484f-ae79-f9274db72eec None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:43.747101 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1777ba82-2b66-46a9-b9e1-98b08047cbda None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:43.781305 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-859db1c6-aeb3-4f30-9cce-b482ae19a691 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:43.781916 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859db1c6-aeb3-4f30-9cce-b482ae19a691 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:43.782196 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859db1c6-aeb3-4f30-9cce-b482ae19a691 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:43.782503 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859db1c6-aeb3-4f30-9cce-b482ae19a691 None admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:43.783452 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-859db1c6-aeb3-4f30-9cce-b482ae19a691 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:43.783762 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-859db1c6-aeb3-4f30-9cce-b482ae19a691 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:43.785233 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-859db1c6-aeb3-4f30-9cce-b482ae19a691 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:43.797324 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 331/673] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Oct 28 09:31:43 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:43.798000 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-859db1c6-aeb3-4f30-9cce-b482ae19a691 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:43.812755 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:43.826308 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe6b18df-a7a6-484f-ae79-f9274db72eec None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:43.860910 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59738df9-b4e3-4e01-b370-684b64d93b27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:43.861548 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59738df9-b4e3-4e01-b370-684b64d93b27 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:43.861833 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59738df9-b4e3-4e01-b370-684b64d93b27 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:43.862113 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59738df9-b4e3-4e01-b370-684b64d93b27 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:43.873328 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59738df9-b4e3-4e01-b370-684b64d93b27 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:31:43.878187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-59738df9-b4e3-4e01-b370-684b64d93b27 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 09:31:43.878187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:43.878187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:43.878187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:43.878187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:43.878187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:43.878187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:43.878187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:43.878187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:43.878187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:43.878187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:43.878187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:43.878187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:31:43.878187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:31:43.878187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:31:43.878187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:31:43.879147 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:31:43.879147 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:31:43.879147 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:31:43.879147 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:31:43.879147 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:43.879147 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:43.879147 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:43.879147 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:43.879147 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:43.879147 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:43.879147 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:43.879147 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:43.879147 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:43.879147 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:43.879147 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:43.879147 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:43.880031 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:43.880031 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:43.880031 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:43.880031 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:43.880031 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:31:43.880031 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:31:43.880031 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:31:43.880031 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:31:43.880031 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:31:43.880031 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:31:43.880031 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 09:31:43.880031 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:43.880031 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 343/674] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Oct 28 09:31:43 2022] GET /identity/v3/roles/service => generated 85 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:43.880718 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59738df9-b4e3-4e01-b370-684b64d93b27 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:43.890292 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:43.902942 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-859db1c6-aeb3-4f30-9cce-b482ae19a691 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:43.936652 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2192f000-065f-439b-b2fa-26e8b16ad27e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:43.937197 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2192f000-065f-439b-b2fa-26e8b16ad27e None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:43.937473 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2192f000-065f-439b-b2fa-26e8b16ad27e None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:43.937745 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2192f000-065f-439b-b2fa-26e8b16ad27e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:43.944778 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2192f000-065f-439b-b2fa-26e8b16ad27e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:43.946211 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2192f000-065f-439b-b2fa-26e8b16ad27e None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:43.957593 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-2192f000-065f-439b-b2fa-26e8b16ad27e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:31:43.957593 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:43.957593 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:43.957593 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:43.957593 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:43.957593 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:43.957593 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:43.957593 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:43.957593 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:43.957593 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:43.957593 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:43.957593 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:43.957593 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 09:31:43.957593 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 09:31:43.957593 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 09:31:43.958848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 09:31:43.958848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:43.958848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:43.958848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:43.958848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:43.958848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:43.958848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:43.958848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:43.958848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:43.958848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:43.958848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:43.958848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:43.958848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:43.958848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:43.958848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:43.958848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:43.958848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:43.959737 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 09:31:43.959737 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 09:31:43.959737 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 09:31:43.959737 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 09:31:43.959737 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 09:31:43.959737 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 09:31:43.959737 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:31:43.959737 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:43.959737 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 332/675] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:31:43 2022] GET /identity/v3/projects/service => generated 88 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:43.960352 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8af99e-73f8-4349-9c80-b005570332bf None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:43.960629 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8af99e-73f8-4349-9c80-b005570332bf None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:43.960899 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8af99e-73f8-4349-9c80-b005570332bf None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:43.962275 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 333/676] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:43.962920 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed8af99e-73f8-4349-9c80-b005570332bf None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:43.983814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59738df9-b4e3-4e01-b370-684b64d93b27 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:44.017693 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46952a10-783d-4e67-9a5a-a8fa9ed85182 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:44.018334 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46952a10-783d-4e67-9a5a-a8fa9ed85182 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:44.018644 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46952a10-783d-4e67-9a5a-a8fa9ed85182 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:44.019070 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46952a10-783d-4e67-9a5a-a8fa9ed85182 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:44.019813 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46952a10-783d-4e67-9a5a-a8fa9ed85182 None admin] RBAC: Adding query filter params (role.id=e5d8d090857a4a018d0ec3c79ece6472, scope.project.id=455d1c826b974bc38497d18a516867c7, user.id=f4b2ea9aa31445d69e0c1d96b5657708) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:44.020127 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46952a10-783d-4e67-9a5a-a8fa9ed85182 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:44.021952 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46952a10-783d-4e67-9a5a-a8fa9ed85182 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:44.028519 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 344/677] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:31:43 2022] GET /identity/v3/role_assignments?user.id=f4b2ea9aa31445d69e0c1d96b5657708&scope.project.id=455d1c826b974bc38497d18a516867c7&role.id=e5d8d090857a4a018d0ec3c79ece6472 => generated 608 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:44.029315 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46952a10-783d-4e67-9a5a-a8fa9ed85182 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:44.049332 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:44.089100 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed8af99e-73f8-4349-9c80-b005570332bf None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:44.114226 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:44.131018 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c829bff1-0b9a-436a-a703-fafe7a3adbf0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:44.131659 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c829bff1-0b9a-436a-a703-fafe7a3adbf0 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:44.132503 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c829bff1-0b9a-436a-a703-fafe7a3adbf0 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:44.132839 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c829bff1-0b9a-436a-a703-fafe7a3adbf0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:44.133944 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c829bff1-0b9a-436a-a703-fafe7a3adbf0 None admin] RBAC: Adding query filter params (role.id=e5d8d090857a4a018d0ec3c79ece6472, scope.project.id=455d1c826b974bc38497d18a516867c7, user.id=1021f1cc73b54d6eaa647a5f49815ef8) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:44.134255 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c829bff1-0b9a-436a-a703-fafe7a3adbf0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:44.135805 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c829bff1-0b9a-436a-a703-fafe7a3adbf0 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:44.142821 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 334/678] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:31:43 2022] GET /identity/v3/role_assignments?user.id=1021f1cc73b54d6eaa647a5f49815ef8&scope.project.id=455d1c826b974bc38497d18a516867c7&role.id=e5d8d090857a4a018d0ec3c79ece6472 => generated 608 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:44.143699 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c829bff1-0b9a-436a-a703-fafe7a3adbf0 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:44.167941 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46952a10-783d-4e67-9a5a-a8fa9ed85182 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:44.215453 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0654a9bd-6b04-40d9-bad4-c5c7542655b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:44.216506 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0654a9bd-6b04-40d9-bad4-c5c7542655b8 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:44.216973 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0654a9bd-6b04-40d9-bad4-c5c7542655b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:44.217429 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0654a9bd-6b04-40d9-bad4-c5c7542655b8 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:44.218304 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0654a9bd-6b04-40d9-bad4-c5c7542655b8 None admin] RBAC: Adding query filter params (name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:44.218872 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0654a9bd-6b04-40d9-bad4-c5c7542655b8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:44.220765 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0654a9bd-6b04-40d9-bad4-c5c7542655b8 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:44.236926 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 345/679] 10.222.0.25 () {56 vars in 1103 bytes} [Fri Oct 28 09:31:44 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:44.240876 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351558a5-2256-446d-a697-f698144413e5 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:44.240876 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351558a5-2256-446d-a697-f698144413e5 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:44.240876 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351558a5-2256-446d-a697-f698144413e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:44.263075 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-351558a5-2256-446d-a697-f698144413e5 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:44.264722 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-351558a5-2256-446d-a697-f698144413e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:44.287233 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c829bff1-0b9a-436a-a703-fafe7a3adbf0 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:44.343581 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-380a7fb0-618b-4d84-bba0-30049e7388f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:44.344985 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380a7fb0-618b-4d84-bba0-30049e7388f6 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:44.344985 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380a7fb0-618b-4d84-bba0-30049e7388f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:44.344985 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-380a7fb0-618b-4d84-bba0-30049e7388f6 None admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:44.345519 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-380a7fb0-618b-4d84-bba0-30049e7388f6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:44.345834 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-380a7fb0-618b-4d84-bba0-30049e7388f6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:44.347384 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-380a7fb0-618b-4d84-bba0-30049e7388f6 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:44.349828 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:44.364763 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 346/680] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:31:44.365971 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 335/681] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Oct 28 09:31:44 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:44.366618 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351558a5-2256-446d-a697-f698144413e5 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:44.377407 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-380a7fb0-618b-4d84-bba0-30049e7388f6 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:44.453093 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:44.503323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351558a5-2256-446d-a697-f698144413e5 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:44.517744 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-380a7fb0-618b-4d84-bba0-30049e7388f6 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:44.545632 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fbb09bf-1fc8-4d36-acea-91ac9ba5ca49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:44.546281 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbb09bf-1fc8-4d36-acea-91ac9ba5ca49 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:44.546667 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbb09bf-1fc8-4d36-acea-91ac9ba5ca49 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:44.546955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fbb09bf-1fc8-4d36-acea-91ac9ba5ca49 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:44.568164 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fbb09bf-1fc8-4d36-acea-91ac9ba5ca49 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:44.571913 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fbb09bf-1fc8-4d36-acea-91ac9ba5ca49 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:44.574270 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8616f8b1-0223-4e2a-8d95-13f4cfd1428d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:44.578162 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8616f8b1-0223-4e2a-8d95-13f4cfd1428d None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:44.578162 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8616f8b1-0223-4e2a-8d95-13f4cfd1428d None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:44.578162 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8616f8b1-0223-4e2a-8d95-13f4cfd1428d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:44.578162 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8616f8b1-0223-4e2a-8d95-13f4cfd1428d None admin] RBAC: Adding query filter params (role.id=e5d8d090857a4a018d0ec3c79ece6472, scope.project.id=455d1c826b974bc38497d18a516867c7, user.id=1db6b98e87e64584932447c82fefc3b7) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:44.578162 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8616f8b1-0223-4e2a-8d95-13f4cfd1428d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:44.578604 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8616f8b1-0223-4e2a-8d95-13f4cfd1428d None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:44.592177 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 336/682] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:31:44 2022] GET /identity/v3/role_assignments?user.id=1db6b98e87e64584932447c82fefc3b7&scope.project.id=455d1c826b974bc38497d18a516867c7&role.id=e5d8d090857a4a018d0ec3c79ece6472 => generated 608 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:44.602607 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-6fbb09bf-1fc8-4d36-acea-91ac9ba5ca49 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:44.602607 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:44.602607 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:44.602607 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:44.602607 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:44.602607 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:44.602607 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:44.602607 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:44.602607 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:44.602607 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:44.602607 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:44.602607 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:44.602607 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:44.602607 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:44.602607 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:44.602607 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:44.603653 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:44.603653 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:44.603653 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:44.603653 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:44.603653 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:44.603653 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:44.603653 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:44.603653 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:44.603653 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:44.603653 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:44.603653 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:44.603653 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:44.603653 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:44.603653 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:44.603653 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:44.603653 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:44.603653 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:44.604618 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:44.604618 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:44.604618 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:44.604618 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 347/683] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:44.611625 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8616f8b1-0223-4e2a-8d95-13f4cfd1428d None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:44.660147 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fbb09bf-1fc8-4d36-acea-91ac9ba5ca49 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:44.696166 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:44.765524 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8616f8b1-0223-4e2a-8d95-13f4cfd1428d None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:44.856646 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fbb09bf-1fc8-4d36-acea-91ac9ba5ca49 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:44.872131 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:44.880727 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7b40f0-5c0c-4cd1-b208-34320e01565c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:44.881397 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7b40f0-5c0c-4cd1-b208-34320e01565c None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:44.882191 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7b40f0-5c0c-4cd1-b208-34320e01565c None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:44.882509 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7b40f0-5c0c-4cd1-b208-34320e01565c None admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:44.883147 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7b40f0-5c0c-4cd1-b208-34320e01565c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:44.883454 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7b40f0-5c0c-4cd1-b208-34320e01565c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:44.884840 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7b40f0-5c0c-4cd1-b208-34320e01565c None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:44.896526 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e93d6db-a46c-4366-9827-7b966cc5702b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:44.897173 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e93d6db-a46c-4366-9827-7b966cc5702b None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:44.897517 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e93d6db-a46c-4366-9827-7b966cc5702b None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:44.897797 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e93d6db-a46c-4366-9827-7b966cc5702b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:44.900585 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 337/684] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:44.905788 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7b40f0-5c0c-4cd1-b208-34320e01565c None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:44.910759 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e93d6db-a46c-4366-9827-7b966cc5702b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:31:44.916970 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-2e93d6db-a46c-4366-9827-7b966cc5702b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:31:44.916970 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:44.916970 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:44.916970 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:44.916970 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:44.916970 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:44.916970 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:44.916970 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:44.916970 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:44.916970 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:44.916970 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:44.916970 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:44.916970 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:31:44.916970 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:31:44.916970 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:31:44.916970 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:31:44.917950 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:31:44.917950 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:31:44.917950 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:31:44.917950 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:31:44.917950 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:44.917950 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:44.917950 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:44.917950 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:44.917950 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:44.917950 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:44.917950 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:44.917950 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:44.917950 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:44.917950 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:44.917950 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:44.917950 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:44.918848 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:44.918848 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:44.918848 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:44.918848 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:44.918848 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:31:44.918848 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:31:44.918848 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:31:44.918848 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:31:44.918848 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:31:44.918848 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:31:44.918848 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:31:44.918848 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:44.918848 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 348/685] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Oct 28 09:31:44 2022] GET /identity/v3/roles/admin => generated 83 bytes in 258 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:31:44.924685 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e93d6db-a46c-4366-9827-7b966cc5702b None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:45.053815 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e93d6db-a46c-4366-9827-7b966cc5702b None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:45.089499 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7b40f0-5c0c-4cd1-b208-34320e01565c None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:45.106542 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d99e3131-a059-409d-ac51-c6008582769e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:45.107171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d99e3131-a059-409d-ac51-c6008582769e None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:45.107450 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d99e3131-a059-409d-ac51-c6008582769e None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:45.107719 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d99e3131-a059-409d-ac51-c6008582769e None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:45.108385 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d99e3131-a059-409d-ac51-c6008582769e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:45.108686 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d99e3131-a059-409d-ac51-c6008582769e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:45.110047 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d99e3131-a059-409d-ac51-c6008582769e None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:45.110820 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:45.120558 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 349/686] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:31:44 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:45.126643 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d99e3131-a059-409d-ac51-c6008582769e None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:45.160578 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad735a1-a098-4f2e-b187-4cd2cd4f2e0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:45.161277 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad735a1-a098-4f2e-b187-4cd2cd4f2e0f None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:45.161564 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad735a1-a098-4f2e-b187-4cd2cd4f2e0f None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:45.161837 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad735a1-a098-4f2e-b187-4cd2cd4f2e0f None admin] PATH_INFO: `/v3/users/swift` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:45.179309 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:45.180871 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad735a1-a098-4f2e-b187-4cd2cd4f2e0f None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:45.182362 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad735a1-a098-4f2e-b187-4cd2cd4f2e0f None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:45.204013 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-dad735a1-a098-4f2e-b187-4cd2cd4f2e0f None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 28 09:31:45.204013 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:45.204013 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:45.204013 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:45.204013 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:45.204013 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:45.204013 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:45.204013 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:45.204013 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:45.204013 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:45.204013 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:45.204013 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:45.204013 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 09:31:45.204013 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 09:31:45.204013 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 09:31:45.204013 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 09:31:45.205033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:45.205033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:45.205033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 09:31:45.205033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:31:45.205033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 09:31:45.205033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:31:45.205033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:45.205033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:45.205033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:45.205033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:45.205033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:45.205033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:45.205033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:45.205033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:45.205033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:45.205033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:45.205033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:45.205033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:45.205970 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:45.205970 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:45.205970 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 09:31:45.205970 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 09:31:45.205970 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 09:31:45.205970 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 09:31:45.205970 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 09:31:45.205970 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 09:31:45.205970 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 28 09:31:45.205970 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:45.210946 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 338/687] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Oct 28 09:31:44 2022] GET /identity/v3/users/swift => generated 83 bytes in 305 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:45.218968 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad735a1-a098-4f2e-b187-4cd2cd4f2e0f None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:45.245695 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d99e3131-a059-409d-ac51-c6008582769e None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:45.288076 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3971893f-b357-4fbb-86cd-dfcb585d19c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:45.289733 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3971893f-b357-4fbb-86cd-dfcb585d19c1 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:45.290848 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3971893f-b357-4fbb-86cd-dfcb585d19c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:45.291707 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3971893f-b357-4fbb-86cd-dfcb585d19c1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:45.317511 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3971893f-b357-4fbb-86cd-dfcb585d19c1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:45.319197 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3971893f-b357-4fbb-86cd-dfcb585d19c1 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:45.342943 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-3971893f-b357-4fbb-86cd-dfcb585d19c1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:45.342943 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:45.342943 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:45.342943 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:45.342943 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:45.342943 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:45.342943 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:45.342943 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:45.342943 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:45.342943 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:45.342943 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:45.342943 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:45.342943 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:45.342943 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:45.342943 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:45.342943 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:45.343982 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:45.343982 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:45.343982 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:45.343982 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:45.343982 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:45.343982 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:45.343982 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:45.343982 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:45.343982 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:45.343982 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:45.343982 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:45.343982 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:45.343982 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:45.343982 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:45.343982 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:45.343982 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:45.343982 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:45.344884 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:45.344884 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:45.344884 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:45.345968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 350/688] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:45.352890 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3971893f-b357-4fbb-86cd-dfcb585d19c1 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:45.421075 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad735a1-a098-4f2e-b187-4cd2cd4f2e0f None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:45.491557 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dadd99d4-b329-4d50-adcb-0d5907ebfd60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:45.492178 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dadd99d4-b329-4d50-adcb-0d5907ebfd60 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:45.492456 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dadd99d4-b329-4d50-adcb-0d5907ebfd60 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:45.492726 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dadd99d4-b329-4d50-adcb-0d5907ebfd60 None admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:45.493463 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dadd99d4-b329-4d50-adcb-0d5907ebfd60 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:45.493765 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dadd99d4-b329-4d50-adcb-0d5907ebfd60 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:45.495225 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dadd99d4-b329-4d50-adcb-0d5907ebfd60 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:45.527835 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 339/689] 10.222.0.25 () {56 vars in 1135 bytes} [Fri Oct 28 09:31:45 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 309 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:45.531895 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dadd99d4-b329-4d50-adcb-0d5907ebfd60 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:45.550256 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:45.585848 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3971893f-b357-4fbb-86cd-dfcb585d19c1 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:45.614027 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:45.655431 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8e79d6-8344-458c-a54e-924a7f72e75f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:45.656070 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8e79d6-8344-458c-a54e-924a7f72e75f None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:45.656340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8e79d6-8344-458c-a54e-924a7f72e75f None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:45.656612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8e79d6-8344-458c-a54e-924a7f72e75f None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:45.657266 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b8e79d6-8344-458c-a54e-924a7f72e75f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:45.657575 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b8e79d6-8344-458c-a54e-924a7f72e75f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:45.659246 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b8e79d6-8344-458c-a54e-924a7f72e75f None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:45.689595 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 351/690] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 338 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:45.697587 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8e79d6-8344-458c-a54e-924a7f72e75f None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:45.736864 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dadd99d4-b329-4d50-adcb-0d5907ebfd60 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:45.789778 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d044c132-f605-43b7-88be-628e691a141e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:45.790411 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d044c132-f605-43b7-88be-628e691a141e None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:45.790975 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d044c132-f605-43b7-88be-628e691a141e None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:45.791253 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d044c132-f605-43b7-88be-628e691a141e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:45.881568 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d044c132-f605-43b7-88be-628e691a141e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:45.883333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d044c132-f605-43b7-88be-628e691a141e None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:45.921968 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:45.938061 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8e79d6-8344-458c-a54e-924a7f72e75f None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:46.008955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3b2f2a-7416-46ed-bf8e-83b9e7de991d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:46.009811 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3b2f2a-7416-46ed-bf8e-83b9e7de991d None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:46.010090 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3b2f2a-7416-46ed-bf8e-83b9e7de991d None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:46.010355 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3b2f2a-7416-46ed-bf8e-83b9e7de991d None admin] PATH_INFO: `/v3/users/nova` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:46.028008 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f3b2f2a-7416-46ed-bf8e-83b9e7de991d None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:46.029466 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f3b2f2a-7416-46ed-bf8e-83b9e7de991d None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:46.030861 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 340/691] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:31:45 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 499 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Oct 28 09:31:46.042252 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d044c132-f605-43b7-88be-628e691a141e None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:46.051940 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-3f3b2f2a-7416-46ed-bf8e-83b9e7de991d None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 28 09:31:46.051940 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:46.051940 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:46.051940 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:46.051940 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:46.051940 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:46.051940 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:46.051940 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:46.051940 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:46.051940 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:46.051940 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:46.051940 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:46.051940 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 09:31:46.051940 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 09:31:46.051940 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 09:31:46.051940 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 09:31:46.052990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:46.052990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:46.052990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 09:31:46.052990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:31:46.052990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 09:31:46.052990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:31:46.052990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:46.052990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:46.052990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:46.052990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:46.052990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:46.052990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:46.052990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:46.052990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:46.052990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:46.052990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:46.052990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:46.052990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:46.054504 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:46.054504 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:46.054504 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 09:31:46.054504 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 09:31:46.054504 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 09:31:46.054504 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 09:31:46.054504 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 09:31:46.054504 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 09:31:46.054504 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 28 09:31:46.054504 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:46.054504 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 352/692] 10.222.0.25 () {56 vars in 1093 bytes} [Fri Oct 28 09:31:45 2022] GET /identity/v3/users/nova => generated 82 bytes in 357 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:46.061247 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3b2f2a-7416-46ed-bf8e-83b9e7de991d None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:46.087299 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:46.191865 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d044c132-f605-43b7-88be-628e691a141e None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:46.242420 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3b2f2a-7416-46ed-bf8e-83b9e7de991d None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:46.280874 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6a84ec-7bcd-40d4-b37a-814508fdda7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:46.281491 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6a84ec-7bcd-40d4-b37a-814508fdda7f None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:46.281772 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6a84ec-7bcd-40d4-b37a-814508fdda7f None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:46.282046 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6a84ec-7bcd-40d4-b37a-814508fdda7f None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:46.284043 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6a84ec-7bcd-40d4-b37a-814508fdda7f None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:46.284358 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6a84ec-7bcd-40d4-b37a-814508fdda7f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:46.285799 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6a84ec-7bcd-40d4-b37a-814508fdda7f None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:46.289366 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e033f5fa-7694-4999-af1f-7684a43f9166 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:46.290010 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e033f5fa-7694-4999-af1f-7684a43f9166 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:46.290287 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e033f5fa-7694-4999-af1f-7684a43f9166 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:46.290619 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e033f5fa-7694-4999-af1f-7684a43f9166 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:46.294974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:46.306408 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e033f5fa-7694-4999-af1f-7684a43f9166 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:46.307898 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e033f5fa-7694-4999-af1f-7684a43f9166 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:46.322351 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 353/693] 10.222.0.25 () {56 vars in 1133 bytes} [Fri Oct 28 09:31:46 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 263 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:46.325662 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-e033f5fa-7694-4999-af1f-7684a43f9166 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:46.325662 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:46.325662 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:46.325662 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:46.325662 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:46.325662 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:46.325662 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:46.325662 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:46.325662 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:46.325662 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:46.325662 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:46.325662 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:46.325662 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:46.325662 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:46.325662 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:46.325662 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:46.326621 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:46.326621 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:46.326621 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:46.326621 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:46.326621 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:46.326621 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:46.326621 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:46.326621 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:46.326621 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:46.326621 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:46.326621 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:46.326621 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:46.326621 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:46.326621 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:46.326621 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:46.326621 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:46.326621 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:46.327529 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:46.327529 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:46.327529 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:46.327529 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 341/694] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 291 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:46.329211 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6a84ec-7bcd-40d4-b37a-814508fdda7f None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:46.335526 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e033f5fa-7694-4999-af1f-7684a43f9166 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:46.470913 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:46.495858 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6a84ec-7bcd-40d4-b37a-814508fdda7f None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:46.535063 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4d9630-6ab9-4850-b1aa-e0b6cb398e7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:46.536395 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4d9630-6ab9-4850-b1aa-e0b6cb398e7f None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:46.536900 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4d9630-6ab9-4850-b1aa-e0b6cb398e7f None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:46.537452 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4d9630-6ab9-4850-b1aa-e0b6cb398e7f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:46.562343 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e033f5fa-7694-4999-af1f-7684a43f9166 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:46.648507 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1157688-13d4-47ad-b531-9ca8d5df88bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:46.650020 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1157688-13d4-47ad-b531-9ca8d5df88bc None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:46.650603 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1157688-13d4-47ad-b531-9ca8d5df88bc None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:46.651085 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1157688-13d4-47ad-b531-9ca8d5df88bc None admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:46.652058 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1157688-13d4-47ad-b531-9ca8d5df88bc None admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:46.653833 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1157688-13d4-47ad-b531-9ca8d5df88bc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:46.655517 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1157688-13d4-47ad-b531-9ca8d5df88bc None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:46.677969 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b4d9630-6ab9-4850-b1aa-e0b6cb398e7f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:46.680508 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b4d9630-6ab9-4850-b1aa-e0b6cb398e7f None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:46.687590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 342/695] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 352 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:46.690474 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1157688-13d4-47ad-b531-9ca8d5df88bc None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:46.787891 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:46.819741 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:46.861504 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 354/696] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:31:46 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 533 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Oct 28 09:31:46.862992 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5a575e-1d0d-4de0-be73-c3b70079b400 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:46.863279 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5a575e-1d0d-4de0-be73-c3b70079b400 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:46.863555 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5a575e-1d0d-4de0-be73-c3b70079b400 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:46.869617 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 355/697] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:46.870528 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f5a575e-1d0d-4de0-be73-c3b70079b400 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:46.873112 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1157688-13d4-47ad-b531-9ca8d5df88bc None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:46.921586 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-100e76e3-6d00-466f-8765-39bfd5ce2e14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:46.922235 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100e76e3-6d00-466f-8765-39bfd5ce2e14 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:46.922816 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100e76e3-6d00-466f-8765-39bfd5ce2e14 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:46.922816 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-100e76e3-6d00-466f-8765-39bfd5ce2e14 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:46.932244 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-100e76e3-6d00-466f-8765-39bfd5ce2e14 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:46.934042 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-100e76e3-6d00-466f-8765-39bfd5ce2e14 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:46.951613 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-100e76e3-6d00-466f-8765-39bfd5ce2e14 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:31:46.951613 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:46.951613 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:46.951613 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:46.951613 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:46.951613 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:46.951613 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:46.951613 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:46.951613 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:46.951613 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:46.951613 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:46.951613 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:46.951613 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 09:31:46.951613 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 09:31:46.951613 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 09:31:46.952556 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 09:31:46.952556 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:46.952556 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:46.952556 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:46.952556 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:46.952556 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:46.952556 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:46.952556 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:46.952556 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:46.952556 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:46.952556 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:46.952556 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:46.952556 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:46.952556 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:46.952556 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:46.952556 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:46.952556 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:46.953471 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 09:31:46.953471 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 09:31:46.953471 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 09:31:46.953471 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 09:31:46.953471 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 09:31:46.953471 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 09:31:46.953471 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:31:46.953471 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:46.953471 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 343/698] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:31:46 2022] GET /identity/v3/projects/service => generated 88 bytes in 263 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:31:46.954463 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59491cc2-44f1-4ac5-9293-8e4d85c03a77 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:46.954734 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59491cc2-44f1-4ac5-9293-8e4d85c03a77 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:46.955008 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59491cc2-44f1-4ac5-9293-8e4d85c03a77 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:46.974184 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-59491cc2-44f1-4ac5-9293-8e4d85c03a77 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:46.976287 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-59491cc2-44f1-4ac5-9293-8e4d85c03a77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:46.997242 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f5a575e-1d0d-4de0-be73-c3b70079b400 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:47.035105 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7458927-1b68-4538-8a2c-8580ceb7fe4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:47.035738 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7458927-1b68-4538-8a2c-8580ceb7fe4b None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:47.036019 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7458927-1b68-4538-8a2c-8580ceb7fe4b None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:47.036290 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7458927-1b68-4538-8a2c-8580ceb7fe4b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:47.047143 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7458927-1b68-4538-8a2c-8580ceb7fe4b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:47.048521 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7458927-1b68-4538-8a2c-8580ceb7fe4b None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:47.056725 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 344/699] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:31:47.058212 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bac18b0-8a55-4f9a-9899-2d5f4c7fb0f8 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:47.058667 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bac18b0-8a55-4f9a-9899-2d5f4c7fb0f8 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:47.058792 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bac18b0-8a55-4f9a-9899-2d5f4c7fb0f8 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:47.062466 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 345/700] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:47.062466 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bac18b0-8a55-4f9a-9899-2d5f4c7fb0f8 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:47.071843 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-a7458927-1b68-4538-8a2c-8580ceb7fe4b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:47.071843 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:47.071843 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:47.071843 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:47.071843 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:47.071843 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:47.071843 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:47.071843 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:47.071843 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:47.071843 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:47.071843 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:47.071843 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:47.071843 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:47.071843 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:47.071843 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:47.071843 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:47.072892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:47.072892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:47.072892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:47.072892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:47.072892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:47.072892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:47.072892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:47.072892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:47.072892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:47.072892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:47.072892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:47.072892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:47.072892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:47.072892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:47.072892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:47.072892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:47.072892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:47.073781 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:47.073781 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:47.073781 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:47.077884 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 356/701] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:47.079354 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183b2e74-e4e4-49c7-a606-3065d6c0bd70 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:47.079688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183b2e74-e4e4-49c7-a606-3065d6c0bd70 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:47.079966 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183b2e74-e4e4-49c7-a606-3065d6c0bd70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:47.099012 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-183b2e74-e4e4-49c7-a606-3065d6c0bd70 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:47.100262 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-183b2e74-e4e4-49c7-a606-3065d6c0bd70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:47.111658 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:47.115687 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:47.176369 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 357/702] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:31:47.177216 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183b2e74-e4e4-49c7-a606-3065d6c0bd70 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:47.187363 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bac18b0-8a55-4f9a-9899-2d5f4c7fb0f8 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:47.224607 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-501d88b2-b4a9-4323-8540-769e37f02d88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:47.226186 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501d88b2-b4a9-4323-8540-769e37f02d88 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:47.226930 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501d88b2-b4a9-4323-8540-769e37f02d88 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:47.227462 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501d88b2-b4a9-4323-8540-769e37f02d88 None admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:47.228372 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-501d88b2-b4a9-4323-8540-769e37f02d88 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:47.228903 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-501d88b2-b4a9-4323-8540-769e37f02d88 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:47.230578 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-501d88b2-b4a9-4323-8540-769e37f02d88 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:47.244207 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 346/703] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Oct 28 09:31:47 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:47.245862 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2080d064-8d01-47fc-afbe-996c0edeeaa4 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:47.246380 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2080d064-8d01-47fc-afbe-996c0edeeaa4 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:47.246851 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2080d064-8d01-47fc-afbe-996c0edeeaa4 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:47.252707 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 347/704] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:47 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:47.253701 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2080d064-8d01-47fc-afbe-996c0edeeaa4 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:47.311658 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183b2e74-e4e4-49c7-a606-3065d6c0bd70 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:47.356723 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a634f6-3152-462a-aade-edb9b5758313 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:47.357329 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a634f6-3152-462a-aade-edb9b5758313 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:47.357605 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a634f6-3152-462a-aade-edb9b5758313 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:47.357886 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a634f6-3152-462a-aade-edb9b5758313 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:47.358564 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a634f6-3152-462a-aade-edb9b5758313 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:47.358873 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a634f6-3152-462a-aade-edb9b5758313 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:47.360233 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a634f6-3152-462a-aade-edb9b5758313 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:47.371730 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:47.377509 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 358/705] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:47.377509 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a740a4-4601-46d6-b547-2668baf30395 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:47.377509 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a740a4-4601-46d6-b547-2668baf30395 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:47.377509 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a740a4-4601-46d6-b547-2668baf30395 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:47.382954 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2080d064-8d01-47fc-afbe-996c0edeeaa4 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:47.393387 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-d0a740a4-4601-46d6-b547-2668baf30395 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:47.394600 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d0a740a4-4601-46d6-b547-2668baf30395 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:47.396832 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:47.420786 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b3f9b0-a9c5-4b58-ab24-70035926d601 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:47.421416 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b3f9b0-a9c5-4b58-ab24-70035926d601 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:47.421695 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b3f9b0-a9c5-4b58-ab24-70035926d601 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:47.421968 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1b3f9b0-a9c5-4b58-ab24-70035926d601 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:47.422715 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1b3f9b0-a9c5-4b58-ab24-70035926d601 None admin] RBAC: Adding query filter params (role.id=e5d8d090857a4a018d0ec3c79ece6472, scope.project.id=455d1c826b974bc38497d18a516867c7, user.id=9c6cffe0d56e45c4b8fb54fbe5df6094) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:47.423010 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1b3f9b0-a9c5-4b58-ab24-70035926d601 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:47.424427 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1b3f9b0-a9c5-4b58-ab24-70035926d601 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:47.431217 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 348/706] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:31:47 2022] GET /identity/v3/role_assignments?user.id=9c6cffe0d56e45c4b8fb54fbe5df6094&scope.project.id=455d1c826b974bc38497d18a516867c7&role.id=e5d8d090857a4a018d0ec3c79ece6472 => generated 608 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:47.441679 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b3f9b0-a9c5-4b58-ab24-70035926d601 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:47.473071 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 359/707] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:31:47.474109 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a740a4-4601-46d6-b547-2668baf30395 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:47.591748 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1b3f9b0-a9c5-4b58-ab24-70035926d601 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:47.613344 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a740a4-4601-46d6-b547-2668baf30395 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:47.640401 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e19516d2-f1cd-476b-a012-f1c6056ac4ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:47.641338 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19516d2-f1cd-476b-a012-f1c6056ac4ec None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:47.641818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19516d2-f1cd-476b-a012-f1c6056ac4ec None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:47.642290 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19516d2-f1cd-476b-a012-f1c6056ac4ec None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:47.654487 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e19516d2-f1cd-476b-a012-f1c6056ac4ec None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:47.656213 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e19516d2-f1cd-476b-a012-f1c6056ac4ec None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:47.661852 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbd2eea-d69c-4df9-9bf3-65c6799ed32b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:47.663274 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbd2eea-d69c-4df9-9bf3-65c6799ed32b None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:47.663658 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbd2eea-d69c-4df9-9bf3-65c6799ed32b None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:47.663941 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbd2eea-d69c-4df9-9bf3-65c6799ed32b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:47.674658 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afbd2eea-d69c-4df9-9bf3-65c6799ed32b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:47.674995 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-e19516d2-f1cd-476b-a012-f1c6056ac4ec None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:47.674995 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:47.674995 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:47.674995 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:47.674995 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:47.674995 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:47.674995 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:47.674995 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:47.674995 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:47.674995 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:47.674995 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:47.674995 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:47.674995 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:47.674995 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:47.674995 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:47.674995 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:47.675921 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:47.675921 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:47.675921 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:47.675921 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:47.675921 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:47.675921 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:47.675921 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:47.675921 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:47.675921 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:47.675921 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:47.675921 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:47.675921 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:47.675921 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:47.675921 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:47.675921 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:47.675921 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:47.675921 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:47.676804 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:47.676804 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:47.676804 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:47.676983 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afbd2eea-d69c-4df9-9bf3-65c6799ed32b None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:47.677061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 349/708] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:47.677528 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e19516d2-f1cd-476b-a012-f1c6056ac4ec None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:47.678276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:47.692300 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-afbd2eea-d69c-4df9-9bf3-65c6799ed32b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:31:47.692300 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:47.692300 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:47.692300 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:47.692300 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:47.692300 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:47.692300 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:47.692300 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:47.692300 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:47.692300 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:47.692300 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:47.692300 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:47.692300 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 09:31:47.692300 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 09:31:47.692300 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 09:31:47.693244 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 09:31:47.693244 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:47.693244 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:47.693244 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:47.693244 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:47.693244 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:47.693244 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:47.693244 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:47.693244 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:47.693244 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:47.693244 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:47.693244 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:47.693244 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:47.693244 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:47.693244 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:47.693244 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:47.693244 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:47.694180 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 09:31:47.694180 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 09:31:47.694180 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 09:31:47.694180 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 09:31:47.694180 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 09:31:47.694180 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 09:31:47.694180 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:31:47.694180 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:47.694180 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 360/709] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:31:47 2022] GET /identity/v3/projects/service => generated 88 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:47.695441 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbd2eea-d69c-4df9-9bf3-65c6799ed32b None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:47.704325 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:47.805878 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e19516d2-f1cd-476b-a012-f1c6056ac4ec None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:47.819094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbd2eea-d69c-4df9-9bf3-65c6799ed32b None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:47.842564 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82cb4f88-6a2f-4bb6-993b-de82215df273 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:47.843211 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82cb4f88-6a2f-4bb6-993b-de82215df273 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:47.843655 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82cb4f88-6a2f-4bb6-993b-de82215df273 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:47.843939 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82cb4f88-6a2f-4bb6-993b-de82215df273 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:47.844531 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82cb4f88-6a2f-4bb6-993b-de82215df273 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:47.845907 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82cb4f88-6a2f-4bb6-993b-de82215df273 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:47.857601 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-82cb4f88-6a2f-4bb6-993b-de82215df273 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 28 09:31:47.857601 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:47.857601 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 09:31:47.857601 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 09:31:47.857601 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 09:31:47.857601 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 09:31:47.857601 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 09:31:47.857601 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:31:47.857601 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 28 09:31:47.857601 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:47.857601 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 09:31:47.857601 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:47.857601 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:47.857601 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:47.857601 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:47.857601 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:47.857601 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:47.858693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:47.858693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:47.858693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:47.858693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:47.858693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:47.858693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:47.858693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 09:31:47.858693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 09:31:47.858693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 09:31:47.858693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 09:31:47.858693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:47.858693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:47.858693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:47.858693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:47.858693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:47.858693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:47.864683 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:47.864683 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:47.864683 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:47.864683 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:47.864683 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:47.864683 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:47.864683 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:47.864683 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:47.864683 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:47.864683 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:47.864683 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 09:31:47.864683 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:31:47.864683 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 28 09:31:47.864683 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:47.865450 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f35a2d-3b99-41cb-bdb1-2ef9c3fde451 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:47.865539 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 350/710] 10.222.0.25 () {56 vars in 1129 bytes} [Fri Oct 28 09:31:47 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:47.865607 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f35a2d-3b99-41cb-bdb1-2ef9c3fde451 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:47.865607 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f35a2d-3b99-41cb-bdb1-2ef9c3fde451 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:47.865765 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82cb4f88-6a2f-4bb6-993b-de82215df273 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:47.865837 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f35a2d-3b99-41cb-bdb1-2ef9c3fde451 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:47.865837 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18f35a2d-3b99-41cb-bdb1-2ef9c3fde451 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:47.865837 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18f35a2d-3b99-41cb-bdb1-2ef9c3fde451 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:47.865837 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18f35a2d-3b99-41cb-bdb1-2ef9c3fde451 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:47.880251 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 361/711] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:47.880251 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f35a2d-3b99-41cb-bdb1-2ef9c3fde451 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:47.939011 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:47.953218 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:47.976168 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82cb4f88-6a2f-4bb6-993b-de82215df273 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:47.988567 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f35a2d-3b99-41cb-bdb1-2ef9c3fde451 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:48.019899 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4c1eb7-b05d-44d4-9b63-28506741e74d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:48.020513 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4c1eb7-b05d-44d4-9b63-28506741e74d None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:48.020796 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4c1eb7-b05d-44d4-9b63-28506741e74d None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:48.021241 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4c1eb7-b05d-44d4-9b63-28506741e74d None admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:48.021812 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d4c1eb7-b05d-44d4-9b63-28506741e74d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:48.022061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d4c1eb7-b05d-44d4-9b63-28506741e74d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:48.023838 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d4c1eb7-b05d-44d4-9b63-28506741e74d None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:48.026766 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c0d68d-beed-47c4-a955-6f6054cd7f7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:48.027419 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c0d68d-beed-47c4-a955-6f6054cd7f7c None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:48.027705 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c0d68d-beed-47c4-a955-6f6054cd7f7c None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:48.027984 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57c0d68d-beed-47c4-a955-6f6054cd7f7c None admin] PATH_INFO: `/v3/services/network` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:48.028573 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57c0d68d-beed-47c4-a955-6f6054cd7f7c None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:48.029936 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57c0d68d-beed-47c4-a955-6f6054cd7f7c None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:48.041262 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 351/712] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Oct 28 09:31:47 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:48.041262 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4c1eb7-b05d-44d4-9b63-28506741e74d None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:48.048210 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-57c0d68d-beed-47c4-a955-6f6054cd7f7c None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 28 09:31:48.048210 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:48.048210 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 09:31:48.048210 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 09:31:48.048210 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 09:31:48.048210 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 09:31:48.048210 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 09:31:48.048210 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:31:48.048210 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 28 09:31:48.048210 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:48.048210 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 09:31:48.048210 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:48.048210 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:48.048210 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:48.048210 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:48.048210 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:48.048210 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:48.049245 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:48.049245 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:48.049245 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:48.049245 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:48.049245 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:48.049245 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:48.049245 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 09:31:48.049245 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 09:31:48.049245 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 09:31:48.049245 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 09:31:48.049245 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:48.049245 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:48.049245 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:48.049245 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:48.049245 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:48.049245 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:48.050092 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:48.050092 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:48.050092 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:48.050092 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:48.050092 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:48.050092 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:48.050092 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:48.050092 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:48.050092 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:48.050092 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:48.050092 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 09:31:48.050092 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:31:48.050092 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 28 09:31:48.050092 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:48.050092 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 362/713] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:31:47 2022] GET /identity/v3/services/network => generated 88 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:48.051419 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c0d68d-beed-47c4-a955-6f6054cd7f7c None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:48.147226 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4c1eb7-b05d-44d4-9b63-28506741e74d None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:48.164256 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57c0d68d-beed-47c4-a955-6f6054cd7f7c None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:48.183289 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553e7218-3503-4997-b452-ba62e584b0f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:48.183902 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553e7218-3503-4997-b452-ba62e584b0f7 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:48.184182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553e7218-3503-4997-b452-ba62e584b0f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:48.184455 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553e7218-3503-4997-b452-ba62e584b0f7 None admin] PATH_INFO: `/v3/services` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:48.185079 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-553e7218-3503-4997-b452-ba62e584b0f7 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:48.185381 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-553e7218-3503-4997-b452-ba62e584b0f7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:48.186771 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-553e7218-3503-4997-b452-ba62e584b0f7 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:48.190560 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:48.191399 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 352/714] 10.222.0.25 () {56 vars in 1124 bytes} [Fri Oct 28 09:31:48 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:48.191987 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553e7218-3503-4997-b452-ba62e584b0f7 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:48.199219 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:48.206734 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ee013f-d5b0-49b2-a107-d9add4648b1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:48.207351 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ee013f-d5b0-49b2-a107-d9add4648b1d None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:48.207630 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ee013f-d5b0-49b2-a107-d9add4648b1d None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:48.207902 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ee013f-d5b0-49b2-a107-d9add4648b1d None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:48.208549 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ee013f-d5b0-49b2-a107-d9add4648b1d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:48.210593 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ee013f-d5b0-49b2-a107-d9add4648b1d None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:48.232333 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 363/715] 10.222.0.25 () {60 vars in 1132 bytes} [Fri Oct 28 09:31:48 2022] POST /identity/v3/users => generated 254 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:31:48.232977 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ee013f-d5b0-49b2-a107-d9add4648b1d None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:48.332833 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553e7218-3503-4997-b452-ba62e584b0f7 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:48.374159 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ee013f-d5b0-49b2-a107-d9add4648b1d None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:48.374742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5407020-e9ff-496f-b58e-cc63ee42d5e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:48.375225 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5407020-e9ff-496f-b58e-cc63ee42d5e0 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:48.375505 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5407020-e9ff-496f-b58e-cc63ee42d5e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:48.375877 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5407020-e9ff-496f-b58e-cc63ee42d5e0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:48.376718 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5407020-e9ff-496f-b58e-cc63ee42d5e0 None admin] RBAC: Adding query filter params (role.id=e13d798a98ea4c3dbb87a236ec683d32, scope.project.id=455d1c826b974bc38497d18a516867c7, user.id=688ceb201d304c2da5373c3cf36c2d79) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:48.377034 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5407020-e9ff-496f-b58e-cc63ee42d5e0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:48.378518 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5407020-e9ff-496f-b58e-cc63ee42d5e0 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:48.385345 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 353/716] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:31:48 2022] GET /identity/v3/role_assignments?user.id=688ceb201d304c2da5373c3cf36c2d79&scope.project.id=455d1c826b974bc38497d18a516867c7&role.id=e13d798a98ea4c3dbb87a236ec683d32 => generated 259 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:48.386310 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5407020-e9ff-496f-b58e-cc63ee42d5e0 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:48.431272 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c679d12-ad0e-4332-b745-8cf0a126191a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:48.432530 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c679d12-ad0e-4332-b745-8cf0a126191a None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:48.432810 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c679d12-ad0e-4332-b745-8cf0a126191a None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:48.433081 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c679d12-ad0e-4332-b745-8cf0a126191a None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:48.433753 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c679d12-ad0e-4332-b745-8cf0a126191a None admin] RBAC: Adding query filter params (name=network) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:48.434058 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c679d12-ad0e-4332-b745-8cf0a126191a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:48.436651 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c679d12-ad0e-4332-b745-8cf0a126191a None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:48.448205 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 364/717] 10.222.0.25 () {56 vars in 1112 bytes} [Fri Oct 28 09:31:48 2022] GET /identity/v3/services?name=network => generated 124 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:48.456059 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c679d12-ad0e-4332-b745-8cf0a126191a None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:48.485747 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:48.539522 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5407020-e9ff-496f-b58e-cc63ee42d5e0 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:48.548589 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:48.584650 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7266b541-4d6a-4c39-8bc7-d677c03293cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:48.585676 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7266b541-4d6a-4c39-8bc7-d677c03293cc None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:48.586163 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7266b541-4d6a-4c39-8bc7-d677c03293cc None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:48.586773 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7266b541-4d6a-4c39-8bc7-d677c03293cc None admin] PATH_INFO: `/v3/services` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:48.587716 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7266b541-4d6a-4c39-8bc7-d677c03293cc None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:48.588235 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7266b541-4d6a-4c39-8bc7-d677c03293cc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:48.589870 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7266b541-4d6a-4c39-8bc7-d677c03293cc None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:48.599656 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 354/718] 10.222.0.25 () {56 vars in 1124 bytes} [Fri Oct 28 09:31:48 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:48.629409 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c679d12-ad0e-4332-b745-8cf0a126191a None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:48.680685 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7db70af-e49b-45ff-abb5-609d405db4cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:48.681371 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7db70af-e49b-45ff-abb5-609d405db4cc None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:48.681600 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7db70af-e49b-45ff-abb5-609d405db4cc None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:48.681880 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7db70af-e49b-45ff-abb5-609d405db4cc None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:48.682565 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7db70af-e49b-45ff-abb5-609d405db4cc None admin] RBAC: Adding query filter params (type=network) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:48.682877 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7db70af-e49b-45ff-abb5-609d405db4cc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:48.684387 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7db70af-e49b-45ff-abb5-609d405db4cc None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:48.691372 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 365/719] 10.222.0.25 () {56 vars in 1112 bytes} [Fri Oct 28 09:31:48 2022] GET /identity/v3/services?type=network => generated 124 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:50.117487 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcde09c-abcc-4869-9157-f0dec7b816e7 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:50.118062 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcde09c-abcc-4869-9157-f0dec7b816e7 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:50.118062 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcde09c-abcc-4869-9157-f0dec7b816e7 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:50.119749 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 355/720] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:50.126308 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc147a2-9b32-4aa4-863e-d2c37cefb682 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:50.126848 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc147a2-9b32-4aa4-863e-d2c37cefb682 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:50.126960 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc147a2-9b32-4aa4-863e-d2c37cefb682 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:50.152171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-0cc147a2-9b32-4aa4-863e-d2c37cefb682 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:50.153353 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-0cc147a2-9b32-4aa4-863e-d2c37cefb682 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:50.239883 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 366/721] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 09:31:50.556416 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fcde09c-abcc-4869-9157-f0dec7b816e7 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:50.711195 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fcde09c-abcc-4869-9157-f0dec7b816e7 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:50.751156 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:50.757769 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d712a6-2e5b-45d5-8283-f137c7d0ac7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:50.758422 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d712a6-2e5b-45d5-8283-f137c7d0ac7e None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:50.758702 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d712a6-2e5b-45d5-8283-f137c7d0ac7e None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:50.758973 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d712a6-2e5b-45d5-8283-f137c7d0ac7e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:50.769234 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d712a6-2e5b-45d5-8283-f137c7d0ac7e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:31:50.774061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-69d712a6-2e5b-45d5-8283-f137c7d0ac7e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:31:50.774061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:50.774061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:50.774061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:50.774061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:50.774061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:50.774061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:50.774061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:50.774061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:50.774061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:50.774061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:50.774061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:50.774061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:31:50.774061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:31:50.774061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:31:50.774061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:31:50.775016 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:31:50.775016 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:31:50.775016 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:31:50.775016 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:31:50.775016 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:50.775016 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:50.775016 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:50.775016 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:50.775016 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:50.775016 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:50.775016 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:50.775016 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:50.775016 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:50.775016 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:50.775016 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:50.775016 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:50.776301 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:50.776301 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:50.776301 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:50.776301 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:50.776301 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:31:50.776301 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:31:50.776301 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:31:50.776301 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:31:50.776301 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:31:50.776301 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:31:50.776301 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:31:50.776301 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:50.776301 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 356/722] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Oct 28 09:31:50 2022] GET /identity/v3/roles/admin => generated 83 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:50.782508 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc147a2-9b32-4aa4-863e-d2c37cefb682 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:50.853614 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:50.963419 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc147a2-9b32-4aa4-863e-d2c37cefb682 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:51.012533 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e3dd89-4c41-4fdb-bdc0-70628793e137 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:51.013163 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e3dd89-4c41-4fdb-bdc0-70628793e137 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:51.013448 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e3dd89-4c41-4fdb-bdc0-70628793e137 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:51.013726 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e3dd89-4c41-4fdb-bdc0-70628793e137 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:51.014429 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e3dd89-4c41-4fdb-bdc0-70628793e137 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:51.014739 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e3dd89-4c41-4fdb-bdc0-70628793e137 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:51.016111 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e3dd89-4c41-4fdb-bdc0-70628793e137 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:51.029738 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 367/723] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:31:50 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:51.035466 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d712a6-2e5b-45d5-8283-f137c7d0ac7e None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:51.051011 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0638633-96c7-44bb-9623-b8296c590db7 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:51.051314 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0638633-96c7-44bb-9623-b8296c590db7 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:51.051599 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0638633-96c7-44bb-9623-b8296c590db7 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:51.053225 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 368/724] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 09:31:51.059137 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb0ce4a9-588e-4cd8-afb9-c2741ab2c315 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:51.059420 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb0ce4a9-588e-4cd8-afb9-c2741ab2c315 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:51.059698 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb0ce4a9-588e-4cd8-afb9-c2741ab2c315 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:51.081404 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-bb0ce4a9-588e-4cd8-afb9-c2741ab2c315 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:51.082701 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-bb0ce4a9-588e-4cd8-afb9-c2741ab2c315 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:51.159354 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 369/725] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Oct 28 09:31:51.244322 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d712a6-2e5b-45d5-8283-f137c7d0ac7e None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:51.301544 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb898d5-4f56-4b05-be27-658ebcf1e412 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:51.302202 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb898d5-4f56-4b05-be27-658ebcf1e412 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:51.302492 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb898d5-4f56-4b05-be27-658ebcf1e412 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:51.302767 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb898d5-4f56-4b05-be27-658ebcf1e412 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:51.321024 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb898d5-4f56-4b05-be27-658ebcf1e412 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:51.322723 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb898d5-4f56-4b05-be27-658ebcf1e412 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:51.338805 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-0cb898d5-4f56-4b05-be27-658ebcf1e412 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:51.338805 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:51.338805 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:51.338805 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:51.338805 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:51.338805 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:51.338805 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:51.338805 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:51.338805 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:51.338805 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:51.338805 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:51.338805 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:51.338805 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:51.338805 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:51.338805 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:51.338805 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:51.341064 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:51.341064 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:51.341064 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:51.341064 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:51.341064 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:51.341064 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:51.341064 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:51.341064 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:51.341064 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:51.341064 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:51.341064 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:51.341064 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:51.341064 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:51.341064 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:51.341064 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:51.341064 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:51.341064 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:51.342759 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:51.342759 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:51.342759 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:51.342759 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 357/726] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 305 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:51.345469 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb0ce4a9-588e-4cd8-afb9-c2741ab2c315 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:51.386825 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:51.391031 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333205f7-3f7a-4dda-9f6a-5811d4c61b42 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:51.391588 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333205f7-3f7a-4dda-9f6a-5811d4c61b42 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:51.392058 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333205f7-3f7a-4dda-9f6a-5811d4c61b42 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:51.393823 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 358/727] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:51 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:51.403763 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c20ff5f-5cdf-4cda-8b2b-41ccd006e3ae None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:51.404365 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c20ff5f-5cdf-4cda-8b2b-41ccd006e3ae None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:51.404848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c20ff5f-5cdf-4cda-8b2b-41ccd006e3ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:51.436138 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-7c20ff5f-5cdf-4cda-8b2b-41ccd006e3ae None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:51.437602 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-7c20ff5f-5cdf-4cda-8b2b-41ccd006e3ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:51.469974 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb0ce4a9-588e-4cd8-afb9-c2741ab2c315 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:51.472028 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:51.506122 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 359/728] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:31:51.507707 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80bfc663-3034-46a3-935f-c8b38c8571a5 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:51.508178 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80bfc663-3034-46a3-935f-c8b38c8571a5 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:51.508632 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80bfc663-3034-46a3-935f-c8b38c8571a5 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:51.509108 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc5e5ff-8953-4333-a77f-bc0c76165442 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:51.509717 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc5e5ff-8953-4333-a77f-bc0c76165442 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:51.509998 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc5e5ff-8953-4333-a77f-bc0c76165442 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:51.514484 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 360/729] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:51.514598 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc5e5ff-8953-4333-a77f-bc0c76165442 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:51.514598 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc5e5ff-8953-4333-a77f-bc0c76165442 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:51.514598 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc5e5ff-8953-4333-a77f-bc0c76165442 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:51.514847 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cde3dac-8f52-4387-bf13-b0392a04b8ea None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:51.514847 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cde3dac-8f52-4387-bf13-b0392a04b8ea None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:51.514847 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cde3dac-8f52-4387-bf13-b0392a04b8ea None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:51.515060 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc5e5ff-8953-4333-a77f-bc0c76165442 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:51.517163 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 361/730] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:51 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:51.517945 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cde3dac-8f52-4387-bf13-b0392a04b8ea None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:51.527060 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 370/731] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:51.528928 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68dd0d8e-f67c-4638-ad25-265e9cfe4d2a None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:51.529219 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68dd0d8e-f67c-4638-ad25-265e9cfe4d2a None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:51.529498 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68dd0d8e-f67c-4638-ad25-265e9cfe4d2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:51.549243 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-68dd0d8e-f67c-4638-ad25-265e9cfe4d2a None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:51.550429 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-68dd0d8e-f67c-4638-ad25-265e9cfe4d2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:51.625347 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 371/732] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:31:51.626841 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4651817-eb61-45b8-a298-498209e2886a None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:51.627130 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4651817-eb61-45b8-a298-498209e2886a None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:51.627406 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4651817-eb61-45b8-a298-498209e2886a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:51.642869 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cde3dac-8f52-4387-bf13-b0392a04b8ea None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:51.650311 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-e4651817-eb61-45b8-a298-498209e2886a None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:51.651632 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e4651817-eb61-45b8-a298-498209e2886a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:51.667142 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:51.686623 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df972a5-05c4-4599-a664-d932f8a3d602 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:51.687572 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df972a5-05c4-4599-a664-d932f8a3d602 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:51.688112 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df972a5-05c4-4599-a664-d932f8a3d602 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:51.688681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df972a5-05c4-4599-a664-d932f8a3d602 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:51.700491 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1df972a5-05c4-4599-a664-d932f8a3d602 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:31:51.705940 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-1df972a5-05c4-4599-a664-d932f8a3d602 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:31:51.705940 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:51.705940 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:51.705940 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:51.705940 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:51.705940 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:51.705940 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:51.705940 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:51.705940 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:51.705940 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:51.705940 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:51.705940 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:51.705940 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:31:51.705940 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:31:51.705940 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:31:51.705940 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:31:51.706959 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:31:51.706959 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:31:51.706959 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:31:51.706959 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:31:51.706959 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:51.706959 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:51.706959 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:51.706959 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:51.706959 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:51.706959 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:51.706959 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:51.706959 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:51.706959 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:51.706959 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:51.706959 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:51.706959 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:51.707807 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:51.707807 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:51.707807 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:51.707807 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:51.707807 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:31:51.707807 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:31:51.707807 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:31:51.707807 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:31:51.707807 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:31:51.707807 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:31:51.707807 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:31:51.707807 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:51.707807 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 362/733] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Oct 28 09:31:51 2022] GET /identity/v3/roles/admin => generated 83 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:51.708604 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df972a5-05c4-4599-a664-d932f8a3d602 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:51.733755 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 372/734] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Oct 28 09:31:51.736471 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4651817-eb61-45b8-a298-498209e2886a None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:51.742942 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:51.818309 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df972a5-05c4-4599-a664-d932f8a3d602 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:51.856113 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4651817-eb61-45b8-a298-498209e2886a None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:51.859528 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e1fc3d-7298-49c9-81f9-ff7ca41139f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:51.859701 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e1fc3d-7298-49c9-81f9-ff7ca41139f0 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:51.859868 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e1fc3d-7298-49c9-81f9-ff7ca41139f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:51.860136 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e1fc3d-7298-49c9-81f9-ff7ca41139f0 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:51.867922 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5e1fc3d-7298-49c9-81f9-ff7ca41139f0 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:51.869370 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5e1fc3d-7298-49c9-81f9-ff7ca41139f0 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:51.881481 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-a5e1fc3d-7298-49c9-81f9-ff7ca41139f0 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 28 09:31:51.881481 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:51.881481 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:51.881481 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:51.881481 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:51.881481 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:51.881481 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:51.881481 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:51.881481 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:51.881481 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:51.881481 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:51.881481 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:51.881481 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 09:31:51.881481 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 09:31:51.881481 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 09:31:51.881481 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 09:31:51.882703 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:51.882703 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:51.882703 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 09:31:51.882703 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:31:51.882703 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 09:31:51.882703 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:31:51.882703 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:51.882703 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:51.882703 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:51.882703 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:51.882703 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:51.882703 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:51.882703 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:51.882703 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:51.882703 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:51.882703 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:51.882703 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:51.882703 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:51.883826 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:51.883826 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:51.883826 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 09:31:51.883826 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 09:31:51.883826 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 09:31:51.883826 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 09:31:51.883826 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 09:31:51.883826 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 09:31:51.883826 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 28 09:31:51.883826 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:51.887305 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 363/735] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Oct 28 09:31:51 2022] GET /identity/v3/users/swift => generated 83 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:51.888119 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e1fc3d-7298-49c9-81f9-ff7ca41139f0 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:51.900005 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd13253-1706-43f1-b9b1-ec116d19fbe6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:51.900924 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd13253-1706-43f1-b9b1-ec116d19fbe6 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:51.901350 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd13253-1706-43f1-b9b1-ec116d19fbe6 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:51.901768 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfd13253-1706-43f1-b9b1-ec116d19fbe6 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:51.902619 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd13253-1706-43f1-b9b1-ec116d19fbe6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:51.903068 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd13253-1706-43f1-b9b1-ec116d19fbe6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:51.904599 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfd13253-1706-43f1-b9b1-ec116d19fbe6 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:51.915362 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 373/736] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:31:51 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:51.916473 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd13253-1706-43f1-b9b1-ec116d19fbe6 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:51.932329 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:52.031103 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfd13253-1706-43f1-b9b1-ec116d19fbe6 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:52.065392 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8fca4d-127f-4afc-853b-9473394491c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:52.065994 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8fca4d-127f-4afc-853b-9473394491c8 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:52.066273 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8fca4d-127f-4afc-853b-9473394491c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:52.066579 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8fca4d-127f-4afc-853b-9473394491c8 None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:52.067292 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee8fca4d-127f-4afc-853b-9473394491c8 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:52.067593 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee8fca4d-127f-4afc-853b-9473394491c8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:52.068987 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee8fca4d-127f-4afc-853b-9473394491c8 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:52.086246 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 374/737] 10.222.0.25 () {56 vars in 1135 bytes} [Fri Oct 28 09:31:51 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:52.087059 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8fca4d-127f-4afc-853b-9473394491c8 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:52.103682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:52.112902 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e1fc3d-7298-49c9-81f9-ff7ca41139f0 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:52.146826 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94780ad2-7205-474b-a9dd-360358a4ea7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:52.147979 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94780ad2-7205-474b-a9dd-360358a4ea7c None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:52.148261 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94780ad2-7205-474b-a9dd-360358a4ea7c None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:52.148536 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94780ad2-7205-474b-a9dd-360358a4ea7c None admin] PATH_INFO: `/v3/roles/service` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:52.151801 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:52.159397 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94780ad2-7205-474b-a9dd-360358a4ea7c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:31:52.164808 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-94780ad2-7205-474b-a9dd-360358a4ea7c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 09:31:52.164808 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:52.164808 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:52.164808 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:52.164808 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:52.164808 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:52.164808 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:52.164808 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:52.164808 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:52.164808 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:52.164808 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:52.164808 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:52.164808 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:31:52.164808 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:31:52.164808 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:31:52.164808 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:31:52.165979 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:31:52.165979 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:31:52.165979 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:31:52.165979 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:31:52.165979 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:52.165979 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:52.165979 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:52.165979 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:52.165979 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:52.165979 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:52.165979 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:52.165979 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:52.165979 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:52.165979 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:52.165979 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:52.165979 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:52.167186 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:52.167186 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:52.167186 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:52.167186 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:52.167186 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:31:52.167186 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:31:52.167186 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:31:52.167186 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:31:52.167186 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:31:52.167186 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:31:52.167186 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 09:31:52.167186 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:52.167186 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 364/738] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Oct 28 09:31:51 2022] GET /identity/v3/roles/service => generated 85 bytes in 278 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:52.168031 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94780ad2-7205-474b-a9dd-360358a4ea7c None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:52.189993 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8fca4d-127f-4afc-853b-9473394491c8 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:52.223772 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7bab2b1-fd97-4537-a420-587c977547c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:52.224403 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7bab2b1-fd97-4537-a420-587c977547c4 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:52.224684 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7bab2b1-fd97-4537-a420-587c977547c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:52.224963 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7bab2b1-fd97-4537-a420-587c977547c4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:52.234732 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7bab2b1-fd97-4537-a420-587c977547c4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:52.236159 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7bab2b1-fd97-4537-a420-587c977547c4 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:52.250825 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-b7bab2b1-fd97-4537-a420-587c977547c4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:52.250825 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:52.250825 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:52.250825 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:52.250825 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:52.250825 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:52.250825 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:52.250825 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:52.250825 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:52.250825 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:52.250825 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:52.250825 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:52.250825 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:52.250825 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:52.250825 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:52.250825 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:52.251782 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:52.251782 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:52.251782 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:52.251782 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:52.251782 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:52.251782 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:52.251782 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:52.251782 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:52.251782 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:52.251782 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:52.251782 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:52.251782 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:52.251782 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:52.251782 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:52.251782 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:52.251782 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:52.251782 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:52.252682 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:52.252682 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:52.252682 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:52.252682 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 375/739] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:52.252682 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7bab2b1-fd97-4537-a420-587c977547c4 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:52.265518 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94780ad2-7205-474b-a9dd-360358a4ea7c None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:52.298232 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7992ead3-f5ed-4179-9c63-34616b22086b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:52.298791 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7992ead3-f5ed-4179-9c63-34616b22086b None admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:52.299061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7992ead3-f5ed-4179-9c63-34616b22086b None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:52.299331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7992ead3-f5ed-4179-9c63-34616b22086b None admin] PATH_INFO: `/v3/services` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:52.299960 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7992ead3-f5ed-4179-9c63-34616b22086b None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:52.301329 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7992ead3-f5ed-4179-9c63-34616b22086b None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:52.316856 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 365/740] 10.222.0.25 () {60 vars in 1141 bytes} [Fri Oct 28 09:31:52 2022] POST /identity/v3/services => generated 251 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:31:52.318041 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7992ead3-f5ed-4179-9c63-34616b22086b None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:52.334233 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:52.371595 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7bab2b1-fd97-4537-a420-587c977547c4 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:52.399952 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:52.430271 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00269f0-92bc-4526-8dbd-f7890dec2fd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:52.430932 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00269f0-92bc-4526-8dbd-f7890dec2fd2 None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:52.431216 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00269f0-92bc-4526-8dbd-f7890dec2fd2 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:52.431540 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00269f0-92bc-4526-8dbd-f7890dec2fd2 None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:52.432220 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c00269f0-92bc-4526-8dbd-f7890dec2fd2 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:52.433449 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7992ead3-f5ed-4179-9c63-34616b22086b None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:52.433627 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c00269f0-92bc-4526-8dbd-f7890dec2fd2 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:52.458459 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 376/741] 10.222.0.25 () {60 vars in 1141 bytes} [Fri Oct 28 09:31:52 2022] POST /identity/v3/services => generated 240 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:31:52.458459 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00269f0-92bc-4526-8dbd-f7890dec2fd2 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:52.492311 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34783c4c-6d4b-46ff-8a20-29237829caba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:52.493333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34783c4c-6d4b-46ff-8a20-29237829caba None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:52.493878 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34783c4c-6d4b-46ff-8a20-29237829caba None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:52.494355 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34783c4c-6d4b-46ff-8a20-29237829caba None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:52.578822 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00269f0-92bc-4526-8dbd-f7890dec2fd2 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:52.583068 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34783c4c-6d4b-46ff-8a20-29237829caba None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:52.584620 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34783c4c-6d4b-46ff-8a20-29237829caba None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:52.597500 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:52.629762 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-486e0a05-f983-44bb-8946-9a5bedb56aff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:52.630388 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486e0a05-f983-44bb-8946-9a5bedb56aff None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:52.630717 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486e0a05-f983-44bb-8946-9a5bedb56aff None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:52.630998 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486e0a05-f983-44bb-8946-9a5bedb56aff None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:52.631743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-486e0a05-f983-44bb-8946-9a5bedb56aff None admin] RBAC: Adding query filter params (name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:52.632055 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-486e0a05-f983-44bb-8946-9a5bedb56aff None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:52.634098 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-486e0a05-f983-44bb-8946-9a5bedb56aff None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:52.644246 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 377/742] 10.222.0.25 () {56 vars in 1103 bytes} [Fri Oct 28 09:31:52 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:52.645300 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-486e0a05-f983-44bb-8946-9a5bedb56aff None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:52.719092 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 366/743] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:31:52 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 401 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 09:31:52.723476 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34783c4c-6d4b-46ff-8a20-29237829caba None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:52.823965 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:52.838945 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34783c4c-6d4b-46ff-8a20-29237829caba None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:52.849692 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:52.880109 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3fc0f8b-8c32-4eaa-a9ee-77913208bb31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:52.880992 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fc0f8b-8c32-4eaa-a9ee-77913208bb31 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:52.881452 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fc0f8b-8c32-4eaa-a9ee-77913208bb31 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:52.882455 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3fc0f8b-8c32-4eaa-a9ee-77913208bb31 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:52.894860 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3fc0f8b-8c32-4eaa-a9ee-77913208bb31 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:52.896561 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3fc0f8b-8c32-4eaa-a9ee-77913208bb31 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:52.917059 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-f3fc0f8b-8c32-4eaa-a9ee-77913208bb31 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:52.917059 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:52.917059 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:52.917059 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:52.917059 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:52.917059 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:52.917059 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:52.917059 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:52.917059 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:52.917059 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:52.917059 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:52.917059 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:52.917059 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:52.917059 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:52.917059 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:52.917059 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:52.918300 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:52.918300 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:52.918300 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:52.918300 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:52.918300 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:52.918300 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:52.918300 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:52.918300 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:52.918300 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:52.918300 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:52.918300 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:52.918300 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:52.918300 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:52.918300 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:52.918300 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:52.918300 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:52.918300 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:52.919398 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:52.919398 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:52.919398 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:52.920174 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-486e0a05-f983-44bb-8946-9a5bedb56aff None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:52.921223 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 367/744] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:52.922271 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3fc0f8b-8c32-4eaa-a9ee-77913208bb31 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:52.966821 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498c1ea6-c030-48c1-bf43-99577fe8bed8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:52.967722 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498c1ea6-c030-48c1-bf43-99577fe8bed8 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:52.968154 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498c1ea6-c030-48c1-bf43-99577fe8bed8 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:52.968574 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498c1ea6-c030-48c1-bf43-99577fe8bed8 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:52.969361 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-498c1ea6-c030-48c1-bf43-99577fe8bed8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:52.969830 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-498c1ea6-c030-48c1-bf43-99577fe8bed8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:52.971606 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-498c1ea6-c030-48c1-bf43-99577fe8bed8 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:52.986902 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 378/745] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 342 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:52.988034 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498c1ea6-c030-48c1-bf43-99577fe8bed8 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:53.054474 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3fc0f8b-8c32-4eaa-a9ee-77913208bb31 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:53.097980 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3ce457-9c3e-4716-bb56-9f0b8fe8c9b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:53.098612 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3ce457-9c3e-4716-bb56-9f0b8fe8c9b0 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:53.098902 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3ce457-9c3e-4716-bb56-9f0b8fe8c9b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:53.099181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f3ce457-9c3e-4716-bb56-9f0b8fe8c9b0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:53.112768 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3ce457-9c3e-4716-bb56-9f0b8fe8c9b0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:53.114437 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f3ce457-9c3e-4716-bb56-9f0b8fe8c9b0 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:53.125277 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:53.132481 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-7f3ce457-9c3e-4716-bb56-9f0b8fe8c9b0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:53.132481 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:53.132481 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:53.132481 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:53.132481 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:53.132481 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:53.132481 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:53.132481 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:53.132481 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:53.132481 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:53.132481 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:53.132481 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:53.132481 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:53.132481 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:53.132481 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:53.132481 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:53.134473 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:53.134473 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:53.134473 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:53.134473 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:53.134473 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:53.134473 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:53.134473 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:53.134473 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:53.134473 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:53.134473 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:53.134473 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:53.134473 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:53.134473 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:53.134473 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:53.134473 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:53.134473 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:53.134473 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:53.136129 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:53.136129 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:53.136129 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:53.136129 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 368/746] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:53.136129 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3ce457-9c3e-4716-bb56-9f0b8fe8c9b0 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:53.138484 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498c1ea6-c030-48c1-bf43-99577fe8bed8 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:53.150616 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:53.208606 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f897e90f-603f-47d0-83d8-659b74c36d69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:53.210637 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f897e90f-603f-47d0-83d8-659b74c36d69 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:53.211412 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f897e90f-603f-47d0-83d8-659b74c36d69 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:53.212330 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f897e90f-603f-47d0-83d8-659b74c36d69 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:53.214914 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f897e90f-603f-47d0-83d8-659b74c36d69 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:53.215703 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f897e90f-603f-47d0-83d8-659b74c36d69 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:53.219141 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f897e90f-603f-47d0-83d8-659b74c36d69 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:53.242737 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 379/747] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:53.244120 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f897e90f-603f-47d0-83d8-659b74c36d69 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:53.250306 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f3ce457-9c3e-4716-bb56-9f0b8fe8c9b0 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:53.289734 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d487789-84e6-431f-ac96-8b8efccdf0d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:53.290698 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d487789-84e6-431f-ac96-8b8efccdf0d4 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:53.290698 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d487789-84e6-431f-ac96-8b8efccdf0d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:53.290923 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d487789-84e6-431f-ac96-8b8efccdf0d4 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:53.299244 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d487789-84e6-431f-ac96-8b8efccdf0d4 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:53.300759 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d487789-84e6-431f-ac96-8b8efccdf0d4 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:53.313242 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-0d487789-84e6-431f-ac96-8b8efccdf0d4 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 28 09:31:53.313242 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:53.313242 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:53.313242 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:53.313242 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:53.313242 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:53.313242 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:53.313242 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:53.313242 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:53.313242 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:53.313242 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:53.313242 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:53.313242 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 09:31:53.313242 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 09:31:53.313242 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 09:31:53.313242 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 09:31:53.314467 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:53.314467 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:53.314467 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 09:31:53.314467 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:31:53.314467 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 09:31:53.314467 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:31:53.314467 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:53.314467 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:53.314467 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:53.314467 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:53.314467 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:53.314467 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:53.314467 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:53.314467 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:53.314467 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:53.314467 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:53.314467 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:53.314467 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:53.315651 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:53.315651 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:53.315651 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 09:31:53.315651 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 09:31:53.315651 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 09:31:53.315651 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 09:31:53.315651 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 09:31:53.315651 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 09:31:53.315651 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 28 09:31:53.315651 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:53.315651 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 369/748] 10.222.0.25 () {56 vars in 1093 bytes} [Fri Oct 28 09:31:53 2022] GET /identity/v3/users/nova => generated 82 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:53.315651 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d487789-84e6-431f-ac96-8b8efccdf0d4 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:53.382078 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f897e90f-603f-47d0-83d8-659b74c36d69 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:53.396380 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:53.429146 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c2f2c3-0e38-43ee-82b0-03db1a8033df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:53.430302 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c2f2c3-0e38-43ee-82b0-03db1a8033df None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:53.430860 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c2f2c3-0e38-43ee-82b0-03db1a8033df None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:53.431410 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c2f2c3-0e38-43ee-82b0-03db1a8033df None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:53.432438 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3c2f2c3-0e38-43ee-82b0-03db1a8033df None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:53.433036 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3c2f2c3-0e38-43ee-82b0-03db1a8033df None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:53.434954 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3c2f2c3-0e38-43ee-82b0-03db1a8033df None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:53.443207 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d487789-84e6-431f-ac96-8b8efccdf0d4 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:53.451473 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 380/749] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:53.452761 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c2f2c3-0e38-43ee-82b0-03db1a8033df None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:53.477942 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:53.493241 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f3dffd-6732-4319-8357-8ada71a5aa09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:53.494151 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f3dffd-6732-4319-8357-8ada71a5aa09 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:53.494672 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f3dffd-6732-4319-8357-8ada71a5aa09 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:53.495157 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f3dffd-6732-4319-8357-8ada71a5aa09 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:53.504662 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35f3dffd-6732-4319-8357-8ada71a5aa09 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:53.506344 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35f3dffd-6732-4319-8357-8ada71a5aa09 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:53.520862 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-35f3dffd-6732-4319-8357-8ada71a5aa09 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 28 09:31:53.520862 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:53.520862 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:53.520862 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:53.520862 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:53.520862 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:53.520862 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:53.520862 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:53.520862 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:53.520862 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:53.520862 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:53.520862 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:53.520862 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 09:31:53.520862 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 09:31:53.520862 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 09:31:53.520862 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 09:31:53.521775 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:53.521775 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:53.521775 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 09:31:53.521775 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:31:53.521775 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 09:31:53.521775 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:31:53.521775 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:53.521775 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:53.521775 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:53.521775 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:53.521775 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:53.521775 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:53.521775 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:53.521775 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:53.521775 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:53.521775 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:53.521775 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:53.521775 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:53.522772 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:53.522772 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:53.522772 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 09:31:53.522772 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 09:31:53.522772 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 09:31:53.522772 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 09:31:53.522772 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 09:31:53.522772 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 09:31:53.522772 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 28 09:31:53.522772 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:53.524242 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 370/750] 10.222.0.25 () {56 vars in 1117 bytes} [Fri Oct 28 09:31:53 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:53.525010 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f3dffd-6732-4319-8357-8ada71a5aa09 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:53.591343 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c2f2c3-0e38-43ee-82b0-03db1a8033df None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:53.626667 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52648c4-4c37-4fb3-bfc4-f87085201b8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:53.627731 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52648c4-4c37-4fb3-bfc4-f87085201b8e None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:53.628233 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52648c4-4c37-4fb3-bfc4-f87085201b8e None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:53.628719 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52648c4-4c37-4fb3-bfc4-f87085201b8e None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:53.629635 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b52648c4-4c37-4fb3-bfc4-f87085201b8e None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:53.630142 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b52648c4-4c37-4fb3-bfc4-f87085201b8e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:53.631910 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b52648c4-4c37-4fb3-bfc4-f87085201b8e None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:53.650691 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 381/751] 10.222.0.25 () {56 vars in 1133 bytes} [Fri Oct 28 09:31:53 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:53.651815 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52648c4-4c37-4fb3-bfc4-f87085201b8e None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:53.653545 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f3dffd-6732-4319-8357-8ada71a5aa09 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:53.690881 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:53.698824 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183a4604-bed6-4392-9f0e-2c51a255fddd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:53.699579 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183a4604-bed6-4392-9f0e-2c51a255fddd None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:53.699889 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183a4604-bed6-4392-9f0e-2c51a255fddd None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:53.700199 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183a4604-bed6-4392-9f0e-2c51a255fddd None admin] PATH_INFO: `/v3/projects/service` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:53.709143 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-183a4604-bed6-4392-9f0e-2c51a255fddd None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:53.710577 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-183a4604-bed6-4392-9f0e-2c51a255fddd None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:53.723420 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-183a4604-bed6-4392-9f0e-2c51a255fddd None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:31:53.723420 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:53.723420 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:53.723420 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:53.723420 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:53.723420 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:53.723420 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:53.723420 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:53.723420 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:53.723420 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:53.723420 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:53.723420 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:53.723420 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 09:31:53.723420 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 09:31:53.723420 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 09:31:53.724331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 09:31:53.724331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:53.724331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:53.724331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:53.724331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:53.724331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:53.724331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:53.724331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:53.724331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:53.724331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:53.724331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:53.724331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:53.724331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:53.724331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:53.724331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:53.724331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:53.724331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:53.725224 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 09:31:53.725224 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 09:31:53.725224 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 09:31:53.725224 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 09:31:53.725224 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 09:31:53.725224 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 09:31:53.725224 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:31:53.725224 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:53.729540 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 371/752] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:31:53 2022] GET /identity/v3/projects/service => generated 88 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:53.731327 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183a4604-bed6-4392-9f0e-2c51a255fddd None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:53.737975 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:53.763200 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52648c4-4c37-4fb3-bfc4-f87085201b8e None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:53.799463 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbcd19aa-d295-4931-9e9b-785e4a3592cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:53.800257 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbcd19aa-d295-4931-9e9b-785e4a3592cb None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:53.800633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbcd19aa-d295-4931-9e9b-785e4a3592cb None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:53.801007 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbcd19aa-d295-4931-9e9b-785e4a3592cb None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:53.801839 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbcd19aa-d295-4931-9e9b-785e4a3592cb None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:53.802248 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbcd19aa-d295-4931-9e9b-785e4a3592cb None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:53.803839 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbcd19aa-d295-4931-9e9b-785e4a3592cb None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:53.826799 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 382/753] 10.222.0.25 () {56 vars in 1149 bytes} [Fri Oct 28 09:31:53 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:53.826799 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbcd19aa-d295-4931-9e9b-785e4a3592cb None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:53.857323 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183a4604-bed6-4392-9f0e-2c51a255fddd None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:53.900305 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b38302c-5d08-43e6-b862-d4cb570f0dde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:53.901772 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b38302c-5d08-43e6-b862-d4cb570f0dde None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:53.902379 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b38302c-5d08-43e6-b862-d4cb570f0dde None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:53.903091 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b38302c-5d08-43e6-b862-d4cb570f0dde None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:53.918853 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b38302c-5d08-43e6-b862-d4cb570f0dde None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:53.920994 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b38302c-5d08-43e6-b862-d4cb570f0dde None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:53.939182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-5b38302c-5d08-43e6-b862-d4cb570f0dde None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:53.939182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:53.939182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:53.939182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:53.939182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:53.939182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:53.939182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:53.939182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:53.939182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:53.939182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:53.939182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:53.939182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:53.939182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:53.939182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:53.939182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:53.939182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:53.940191 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:53.940191 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:53.940191 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:53.940191 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:53.940191 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:53.940191 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:53.940191 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:53.940191 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:53.940191 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:53.940191 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:53.940191 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:53.940191 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:53.940191 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:53.940191 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:53.940191 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:53.940191 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:53.940191 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:53.941095 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:53.941095 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:53.941095 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:53.943274 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 372/754] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:53.944849 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b38302c-5d08-43e6-b862-d4cb570f0dde None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:53.945379 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbcd19aa-d295-4931-9e9b-785e4a3592cb None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:53.980814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-863505bb-7b6f-4266-91c4-68996c99865b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:53.981489 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863505bb-7b6f-4266-91c4-68996c99865b None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:53.981774 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863505bb-7b6f-4266-91c4-68996c99865b None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:53.982045 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863505bb-7b6f-4266-91c4-68996c99865b None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:53.982961 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-863505bb-7b6f-4266-91c4-68996c99865b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:53.983093 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-863505bb-7b6f-4266-91c4-68996c99865b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:53.984606 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-863505bb-7b6f-4266-91c4-68996c99865b None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:53.986235 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:54.003116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 383/755] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Oct 28 09:31:53 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:54.003116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-863505bb-7b6f-4266-91c4-68996c99865b None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:54.011311 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:54.078415 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b38302c-5d08-43e6-b862-d4cb570f0dde None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:54.125583 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8948ed2-2ea9-44b3-9048-361e9c3bd829 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:54.126207 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8948ed2-2ea9-44b3-9048-361e9c3bd829 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:54.126534 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8948ed2-2ea9-44b3-9048-361e9c3bd829 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:54.126812 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8948ed2-2ea9-44b3-9048-361e9c3bd829 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:54.162075 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-863505bb-7b6f-4266-91c4-68996c99865b None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:54.214153 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ccb070-2987-4450-9650-617a74029cc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:54.214856 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8948ed2-2ea9-44b3-9048-361e9c3bd829 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:54.215746 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ccb070-2987-4450-9650-617a74029cc8 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:54.216130 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8948ed2-2ea9-44b3-9048-361e9c3bd829 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:54.216463 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ccb070-2987-4450-9650-617a74029cc8 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:54.216949 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ccb070-2987-4450-9650-617a74029cc8 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:54.217791 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77ccb070-2987-4450-9650-617a74029cc8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:54.218481 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77ccb070-2987-4450-9650-617a74029cc8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:54.220900 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77ccb070-2987-4450-9650-617a74029cc8 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:54.236492 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 384/756] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:54.237999 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ccb070-2987-4450-9650-617a74029cc8 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:54.269535 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:54.310451 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:54.345930 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 373/757] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:31:53 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 398 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 09:31:54.345930 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8948ed2-2ea9-44b3-9048-361e9c3bd829 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:54.375875 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ccb070-2987-4450-9650-617a74029cc8 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:54.424356 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b597bf2-490e-409f-86f5-c19bf9443190 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:54.425290 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b597bf2-490e-409f-86f5-c19bf9443190 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:54.425773 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b597bf2-490e-409f-86f5-c19bf9443190 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:54.426294 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b597bf2-490e-409f-86f5-c19bf9443190 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:54.427257 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b597bf2-490e-409f-86f5-c19bf9443190 None admin] RBAC: Adding query filter params (role.id=e13d798a98ea4c3dbb87a236ec683d32, scope.project.id=455d1c826b974bc38497d18a516867c7, user.id=9c6cffe0d56e45c4b8fb54fbe5df6094) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:54.427786 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b597bf2-490e-409f-86f5-c19bf9443190 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:54.430191 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b597bf2-490e-409f-86f5-c19bf9443190 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:54.446586 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 385/758] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:31:54 2022] GET /identity/v3/role_assignments?user.id=9c6cffe0d56e45c4b8fb54fbe5df6094&scope.project.id=455d1c826b974bc38497d18a516867c7&role.id=e13d798a98ea4c3dbb87a236ec683d32 => generated 259 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:54.447883 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b597bf2-490e-409f-86f5-c19bf9443190 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:54.465898 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8948ed2-2ea9-44b3-9048-361e9c3bd829 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:54.511790 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a20338b-1979-4d09-b3b0-69993d40a6ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:54.511790 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a20338b-1979-4d09-b3b0-69993d40a6ca None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:54.511790 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a20338b-1979-4d09-b3b0-69993d40a6ca None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:54.511790 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a20338b-1979-4d09-b3b0-69993d40a6ca None admin] PATH_INFO: `/v3/projects/service` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:54.520495 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a20338b-1979-4d09-b3b0-69993d40a6ca None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:54.522645 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a20338b-1979-4d09-b3b0-69993d40a6ca None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:54.536890 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-7a20338b-1979-4d09-b3b0-69993d40a6ca None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:31:54.536890 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:54.536890 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:54.536890 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:54.536890 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:54.536890 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:54.536890 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:54.536890 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:54.536890 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:54.536890 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:54.536890 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:54.536890 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:54.536890 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 09:31:54.536890 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 09:31:54.536890 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 09:31:54.538547 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 09:31:54.538547 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:54.538547 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:54.538547 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:54.538547 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:54.538547 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:54.538547 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:54.538547 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:54.538547 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:54.538547 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:54.538547 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:54.538547 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:54.538547 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:54.538547 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:54.538547 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:54.538547 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:54.538547 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:54.539509 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 09:31:54.539509 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 09:31:54.539509 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 09:31:54.539509 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 09:31:54.539509 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 09:31:54.539509 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 09:31:54.539509 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:31:54.539509 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:54.539509 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 374/759] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:31:54 2022] GET /identity/v3/projects/service => generated 88 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:54.544380 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a20338b-1979-4d09-b3b0-69993d40a6ca None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:54.554677 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:54.630416 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b597bf2-490e-409f-86f5-c19bf9443190 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:54.653689 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:54.674837 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d551092c-c69b-4d5f-8918-06913d8efc80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:54.675698 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d551092c-c69b-4d5f-8918-06913d8efc80 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:54.675978 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d551092c-c69b-4d5f-8918-06913d8efc80 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:54.676260 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d551092c-c69b-4d5f-8918-06913d8efc80 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:54.692250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d551092c-c69b-4d5f-8918-06913d8efc80 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:54.693715 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d551092c-c69b-4d5f-8918-06913d8efc80 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:54.713131 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-d551092c-c69b-4d5f-8918-06913d8efc80 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:54.713131 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:54.713131 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:54.713131 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:54.713131 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:54.713131 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:54.713131 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:54.713131 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:54.713131 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:54.713131 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:54.713131 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:54.713131 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:54.713131 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:54.713131 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:54.713131 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:54.713131 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:54.714148 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:54.714148 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:54.714148 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:54.714148 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:54.714148 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:54.714148 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:54.714148 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:54.714148 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:54.714148 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:54.714148 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:54.714148 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:54.714148 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:54.714148 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:54.714148 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:54.714148 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:54.714148 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:54.714148 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:54.715061 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:54.715061 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:54.715061 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:54.719009 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 386/760] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 268 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:54.719009 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd58421-1b72-45be-b2cd-15ca51f5ff78 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:54.719009 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd58421-1b72-45be-b2cd-15ca51f5ff78 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:54.719009 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd58421-1b72-45be-b2cd-15ca51f5ff78 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:54.719669 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 387/761] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:54.720105 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd58421-1b72-45be-b2cd-15ca51f5ff78 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:54.756203 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a20338b-1979-4d09-b3b0-69993d40a6ca None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:54.796910 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f814c50-0adc-493f-8ddb-3d59904e4460 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:54.797523 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f814c50-0adc-493f-8ddb-3d59904e4460 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:54.797801 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f814c50-0adc-493f-8ddb-3d59904e4460 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:54.798072 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f814c50-0adc-493f-8ddb-3d59904e4460 None admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:54.798837 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f814c50-0adc-493f-8ddb-3d59904e4460 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:54.799187 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f814c50-0adc-493f-8ddb-3d59904e4460 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:54.800600 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f814c50-0adc-493f-8ddb-3d59904e4460 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:54.813334 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 375/762] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Oct 28 09:31:54 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:54.818135 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730bd380-0e67-4ef3-98b8-691b6587cfe2 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:54.818135 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730bd380-0e67-4ef3-98b8-691b6587cfe2 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:54.818135 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730bd380-0e67-4ef3-98b8-691b6587cfe2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:54.833488 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-730bd380-0e67-4ef3-98b8-691b6587cfe2 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:54.834769 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-730bd380-0e67-4ef3-98b8-691b6587cfe2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:54.856394 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd58421-1b72-45be-b2cd-15ca51f5ff78 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:54.896540 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb83683c-f0c1-40e7-8606-7c0d20b0c39a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:54.897670 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb83683c-f0c1-40e7-8606-7c0d20b0c39a None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:54.897994 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb83683c-f0c1-40e7-8606-7c0d20b0c39a None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:54.898275 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb83683c-f0c1-40e7-8606-7c0d20b0c39a None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:54.898950 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb83683c-f0c1-40e7-8606-7c0d20b0c39a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:54.899256 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb83683c-f0c1-40e7-8606-7c0d20b0c39a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:54.900618 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb83683c-f0c1-40e7-8606-7c0d20b0c39a None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:54.902303 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:54.917736 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 388/763] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:54.917736 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3d7b9c-42f8-484b-9c0c-73d7493ff795 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:54.917736 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3d7b9c-42f8-484b-9c0c-73d7493ff795 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:54.917736 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3d7b9c-42f8-484b-9c0c-73d7493ff795 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:54.917736 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 389/764] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:54.920103 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e3d7b9c-42f8-484b-9c0c-73d7493ff795 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:54.924714 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 376/765] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:54 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:31:54.924714 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730bd380-0e67-4ef3-98b8-691b6587cfe2 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:54.956297 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:55.039289 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e3d7b9c-42f8-484b-9c0c-73d7493ff795 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:55.042998 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730bd380-0e67-4ef3-98b8-691b6587cfe2 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:55.080339 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f1f30d4-c04e-4985-9a73-5f52ee3c3860 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:55.081431 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1f30d4-c04e-4985-9a73-5f52ee3c3860 None admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:55.081967 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1f30d4-c04e-4985-9a73-5f52ee3c3860 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:55.082567 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f1f30d4-c04e-4985-9a73-5f52ee3c3860 None admin] PATH_INFO: `/v3/projects/455d1c826b974bc38497d18a516867c7/users/688ceb201d304c2da5373c3cf36c2d79/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:55.082931 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fbbaf7c-fb63-4883-a9c4-5184201bc83e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:55.083878 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fbbaf7c-fb63-4883-a9c4-5184201bc83e None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:55.084305 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fbbaf7c-fb63-4883-a9c4-5184201bc83e None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:55.085320 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fbbaf7c-fb63-4883-a9c4-5184201bc83e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:55.086146 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f1f30d4-c04e-4985-9a73-5f52ee3c3860 None admin] RBAC: Authorizing `identity:create_grant(project_id=455d1c826b974bc38497d18a516867c7, user_id=688ceb201d304c2da5373c3cf36c2d79, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:55.088284 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f1f30d4-c04e-4985-9a73-5f52ee3c3860 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:55.093795 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fbbaf7c-fb63-4883-a9c4-5184201bc83e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:55.095723 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fbbaf7c-fb63-4883-a9c4-5184201bc83e None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:55.109677 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 390/766] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Oct 28 09:31:54 2022] PUT /identity/v3/projects/455d1c826b974bc38497d18a516867c7/users/688ceb201d304c2da5373c3cf36c2d79/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:31:55.109677 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb766df-59ba-41d5-8a0a-9f32c64f8f24 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:55.109677 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb766df-59ba-41d5-8a0a-9f32c64f8f24 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:55.109677 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb766df-59ba-41d5-8a0a-9f32c64f8f24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:55.116319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-1fbbaf7c-fb63-4883-a9c4-5184201bc83e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:31:55.116319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:55.116319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:55.116319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:55.116319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:55.116319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:55.116319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:55.116319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:55.116319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:55.116319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:55.116319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:55.116319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:55.116319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 09:31:55.116319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 09:31:55.116319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 09:31:55.117377 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 09:31:55.117377 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:55.117377 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:55.117377 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:55.117377 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:55.117377 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:55.117377 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:55.117377 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:55.117377 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:55.117377 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:55.117377 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:55.117377 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:55.117377 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:55.117377 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:55.117377 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:55.117377 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:55.117377 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:55.118580 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 09:31:55.118580 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 09:31:55.118580 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 09:31:55.118580 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 09:31:55.118580 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 09:31:55.118580 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 09:31:55.118580 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:31:55.118580 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:55.127355 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 377/767] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:31:54 2022] GET /identity/v3/projects/service => generated 88 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:55.131069 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-fbb766df-59ba-41d5-8a0a-9f32c64f8f24 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:55.132527 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-fbb766df-59ba-41d5-8a0a-9f32c64f8f24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:55.133662 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fbbaf7c-fb63-4883-a9c4-5184201bc83e None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:55.222558 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 391/768] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:55 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:31:55.224208 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb766df-59ba-41d5-8a0a-9f32c64f8f24 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:55.255135 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:55.283261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:55.317210 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fbbaf7c-fb63-4883-a9c4-5184201bc83e None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:55.356555 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e53e2e-a905-472d-9a41-e263692fe5a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:55.357219 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e53e2e-a905-472d-9a41-e263692fe5a9 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:55.357857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e53e2e-a905-472d-9a41-e263692fe5a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:55.358136 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07e53e2e-a905-472d-9a41-e263692fe5a9 None admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:55.359187 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e53e2e-a905-472d-9a41-e263692fe5a9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:55.359504 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e53e2e-a905-472d-9a41-e263692fe5a9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:55.361420 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07e53e2e-a905-472d-9a41-e263692fe5a9 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:55.380653 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 378/769] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Oct 28 09:31:55 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:31:55.385159 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e53e2e-a905-472d-9a41-e263692fe5a9 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:55.441989 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb766df-59ba-41d5-8a0a-9f32c64f8f24 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:55.482159 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564c41ba-7090-4320-861f-31e80046deff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:55.483328 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564c41ba-7090-4320-861f-31e80046deff None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:55.483814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564c41ba-7090-4320-861f-31e80046deff None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:55.484286 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564c41ba-7090-4320-861f-31e80046deff None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:55.485070 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564c41ba-7090-4320-861f-31e80046deff None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:55.486846 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564c41ba-7090-4320-861f-31e80046deff None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:55.497475 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-564c41ba-7090-4320-861f-31e80046deff None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 28 09:31:55.497475 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:55.497475 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 09:31:55.497475 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 09:31:55.497475 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 09:31:55.497475 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 09:31:55.497475 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 09:31:55.497475 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:31:55.497475 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 28 09:31:55.497475 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:55.497475 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 09:31:55.497475 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:55.497475 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:55.497475 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:55.497475 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:55.497475 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:55.497475 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:55.498537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:55.498537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:55.498537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:55.498537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:55.498537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:55.498537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:55.498537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 09:31:55.498537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 09:31:55.498537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 09:31:55.498537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 09:31:55.498537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:55.498537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:55.498537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:55.498537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:55.498537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:55.498537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:55.499391 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:55.499391 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:55.499391 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:55.499391 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:55.499391 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:55.499391 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:55.499391 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:55.499391 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:55.499391 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:55.499391 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:55.499391 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 09:31:55.499391 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:31:55.499391 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 28 09:31:55.499391 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:55.502263 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 392/770] 10.222.0.25 () {56 vars in 1129 bytes} [Fri Oct 28 09:31:55 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 278 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:55.514434 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07e53e2e-a905-472d-9a41-e263692fe5a9 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:55.515774 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564c41ba-7090-4320-861f-31e80046deff None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:55.522119 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:55.549645 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591bf660-b1ff-4da0-b11a-055e22ca6f26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:55.550272 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591bf660-b1ff-4da0-b11a-055e22ca6f26 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:55.550593 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591bf660-b1ff-4da0-b11a-055e22ca6f26 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:55.550875 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-591bf660-b1ff-4da0-b11a-055e22ca6f26 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:55.552399 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-591bf660-b1ff-4da0-b11a-055e22ca6f26 None admin] RBAC: Adding query filter params (role.id=e5d8d090857a4a018d0ec3c79ece6472, scope.project.id=455d1c826b974bc38497d18a516867c7, user.id=97c8877c74194a6bbbf25b3c8f5b7dbd) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:55.552711 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-591bf660-b1ff-4da0-b11a-055e22ca6f26 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:55.554636 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-591bf660-b1ff-4da0-b11a-055e22ca6f26 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:55.570427 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 379/771] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:31:55 2022] GET /identity/v3/role_assignments?user.id=97c8877c74194a6bbbf25b3c8f5b7dbd&scope.project.id=455d1c826b974bc38497d18a516867c7&role.id=e5d8d090857a4a018d0ec3c79ece6472 => generated 259 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:55.570427 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591bf660-b1ff-4da0-b11a-055e22ca6f26 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:55.687270 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:55.721167 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564c41ba-7090-4320-861f-31e80046deff None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:55.772774 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-591bf660-b1ff-4da0-b11a-055e22ca6f26 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:55.784078 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0d6b93-4d24-429f-af3f-6460758dd8f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:55.784709 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0d6b93-4d24-429f-af3f-6460758dd8f3 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:55.784988 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0d6b93-4d24-429f-af3f-6460758dd8f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:55.785260 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0d6b93-4d24-429f-af3f-6460758dd8f3 None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:55.785902 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc0d6b93-4d24-429f-af3f-6460758dd8f3 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:55.786209 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc0d6b93-4d24-429f-af3f-6460758dd8f3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:55.787630 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc0d6b93-4d24-429f-af3f-6460758dd8f3 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:55.793488 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 393/772] 10.222.0.25 () {56 vars in 1124 bytes} [Fri Oct 28 09:31:55 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:55.799503 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0d6b93-4d24-429f-af3f-6460758dd8f3 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:55.825933 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8417ab-5d5d-42fd-a019-d30f6574fc7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:55.826607 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8417ab-5d5d-42fd-a019-d30f6574fc7a None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:55.826889 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8417ab-5d5d-42fd-a019-d30f6574fc7a None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:55.827163 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8417ab-5d5d-42fd-a019-d30f6574fc7a None admin] PATH_INFO: `/v3/services/network` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:55.828349 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8417ab-5d5d-42fd-a019-d30f6574fc7a None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:55.829739 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8417ab-5d5d-42fd-a019-d30f6574fc7a None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:55.851943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-8b8417ab-5d5d-42fd-a019-d30f6574fc7a None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 28 09:31:55.851943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:55.851943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 09:31:55.851943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 09:31:55.851943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 09:31:55.851943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 09:31:55.851943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 09:31:55.851943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:31:55.851943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 28 09:31:55.851943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:55.851943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 09:31:55.851943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:55.851943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:55.851943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:55.851943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:55.851943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:55.851943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:55.852983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:55.852983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:55.852983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:55.852983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:55.852983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:55.852983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:55.852983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 09:31:55.852983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 09:31:55.852983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 09:31:55.852983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 09:31:55.852983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:55.852983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:55.852983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:55.852983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:55.852983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:55.852983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:55.853820 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:55.853820 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:55.853820 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:55.853820 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:55.853820 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:55.853820 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:55.853820 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:55.853820 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:55.853820 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:55.853820 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:55.853820 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 09:31:55.853820 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:31:55.853820 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 28 09:31:55.853820 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:55.858442 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 380/773] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:31:55 2022] GET /identity/v3/services/network => generated 88 bytes in 291 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:55.860183 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8417ab-5d5d-42fd-a019-d30f6574fc7a None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:55.938332 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0d6b93-4d24-429f-af3f-6460758dd8f3 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:55.954918 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:55.980052 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f8d77a-7979-4895-9f9a-5b1a3f79752d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:55.980971 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f8d77a-7979-4895-9f9a-5b1a3f79752d None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:55.981461 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f8d77a-7979-4895-9f9a-5b1a3f79752d None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:55.981941 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f8d77a-7979-4895-9f9a-5b1a3f79752d None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:55.983648 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f8d77a-7979-4895-9f9a-5b1a3f79752d None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:55.984409 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f8d77a-7979-4895-9f9a-5b1a3f79752d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:55.986908 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f8d77a-7979-4895-9f9a-5b1a3f79752d None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:55.998167 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 394/774] 10.222.0.25 () {56 vars in 1124 bytes} [Fri Oct 28 09:31:55 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:31:56.005209 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f8d77a-7979-4895-9f9a-5b1a3f79752d None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:56.036856 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:56.148445 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8417ab-5d5d-42fd-a019-d30f6574fc7a None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:56.261701 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3059f25f-78fa-4c0c-9d74-b00af62a8420 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:56.262691 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3059f25f-78fa-4c0c-9d74-b00af62a8420 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:56.263166 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3059f25f-78fa-4c0c-9d74-b00af62a8420 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:56.263631 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3059f25f-78fa-4c0c-9d74-b00af62a8420 None admin] PATH_INFO: `/v3/services` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:56.274564 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3059f25f-78fa-4c0c-9d74-b00af62a8420 None admin] RBAC: Adding query filter params (name=network) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:56.274959 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3059f25f-78fa-4c0c-9d74-b00af62a8420 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:56.276390 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3059f25f-78fa-4c0c-9d74-b00af62a8420 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:56.282106 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f8d77a-7979-4895-9f9a-5b1a3f79752d None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:56.292790 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 381/775] 10.222.0.25 () {56 vars in 1112 bytes} [Fri Oct 28 09:31:55 2022] GET /identity/v3/services?name=network => generated 124 bytes in 433 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:56.300689 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3059f25f-78fa-4c0c-9d74-b00af62a8420 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:56.350380 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49bae56-306d-48c6-863e-54ba15448c11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:56.352151 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49bae56-306d-48c6-863e-54ba15448c11 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:56.352883 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49bae56-306d-48c6-863e-54ba15448c11 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:56.353584 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49bae56-306d-48c6-863e-54ba15448c11 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:56.355105 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f49bae56-306d-48c6-863e-54ba15448c11 None admin] RBAC: Adding query filter params (interface=public, service_id=2e2c21ce49914653bb5049b1608a7fa0, region_id=RegionOne) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:56.356413 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f49bae56-306d-48c6-863e-54ba15448c11 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:56.359131 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f49bae56-306d-48c6-863e-54ba15448c11 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:56.375353 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 395/776] 10.222.0.25 () {56 vars in 1251 bytes} [Fri Oct 28 09:31:56 2022] GET /identity/v3/endpoints?service_id=2e2c21ce49914653bb5049b1608a7fa0&interface=public®ion_id=RegionOne => generated 194 bytes in 372 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:56.386375 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49bae56-306d-48c6-863e-54ba15448c11 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:56.521393 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3059f25f-78fa-4c0c-9d74-b00af62a8420 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:56.537687 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:56.592707 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57419c7-5c92-4825-a9e2-e38ab003e5ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:56.593875 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57419c7-5c92-4825-a9e2-e38ab003e5ee None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:56.594362 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57419c7-5c92-4825-a9e2-e38ab003e5ee None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:56.594865 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57419c7-5c92-4825-a9e2-e38ab003e5ee None admin] PATH_INFO: `/v3/services` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:56.595987 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57419c7-5c92-4825-a9e2-e38ab003e5ee None admin] RBAC: Adding query filter params (type=network) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:56.596518 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57419c7-5c92-4825-a9e2-e38ab003e5ee None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:56.599188 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57419c7-5c92-4825-a9e2-e38ab003e5ee None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:56.611582 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 382/777] 10.222.0.25 () {56 vars in 1112 bytes} [Fri Oct 28 09:31:56 2022] GET /identity/v3/services?type=network => generated 350 bytes in 311 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:56.621782 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:56.623772 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57419c7-5c92-4825-a9e2-e38ab003e5ee None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:56.671822 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49bae56-306d-48c6-863e-54ba15448c11 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:56.752979 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b15800-bfd7-452d-96c1-c6dd7e98997e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:56.753965 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b15800-bfd7-452d-96c1-c6dd7e98997e None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:56.755670 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b15800-bfd7-452d-96c1-c6dd7e98997e None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:56.756149 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b15800-bfd7-452d-96c1-c6dd7e98997e None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:56.757023 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5b15800-bfd7-452d-96c1-c6dd7e98997e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:56.759602 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5b15800-bfd7-452d-96c1-c6dd7e98997e None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:56.771430 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 396/778] 10.222.0.25 () {56 vars in 1087 bytes} [Fri Oct 28 09:31:56 2022] GET /identity/v3/services => generated 772 bytes in 387 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:56.809555 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57419c7-5c92-4825-a9e2-e38ab003e5ee None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:56.854271 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c269682-d562-4fad-8cd1-3e79cf61f934 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:56.855287 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c269682-d562-4fad-8cd1-3e79cf61f934 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:56.855813 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c269682-d562-4fad-8cd1-3e79cf61f934 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:56.856281 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c269682-d562-4fad-8cd1-3e79cf61f934 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:56.857185 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c269682-d562-4fad-8cd1-3e79cf61f934 None admin] RBAC: Adding query filter params (interface=public, service_id=4521ec7bd9274bfc99d2638c1c962034, region_id=RegionOne) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:56.857689 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c269682-d562-4fad-8cd1-3e79cf61f934 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:56.859277 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c269682-d562-4fad-8cd1-3e79cf61f934 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:56.868577 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 383/779] 10.222.0.25 () {56 vars in 1251 bytes} [Fri Oct 28 09:31:56 2022] GET /identity/v3/endpoints?service_id=4521ec7bd9274bfc99d2638c1c962034&interface=public®ion_id=RegionOne => generated 194 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:56.877371 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b15800-bfd7-452d-96c1-c6dd7e98997e None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:57.109319 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b15800-bfd7-452d-96c1-c6dd7e98997e None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:57.171297 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7910af9-140e-49fa-b4ba-2fd44eb2c2f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:57.172214 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7910af9-140e-49fa-b4ba-2fd44eb2c2f8 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:57.172647 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7910af9-140e-49fa-b4ba-2fd44eb2c2f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:57.173070 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7910af9-140e-49fa-b4ba-2fd44eb2c2f8 None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:57.174008 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7910af9-140e-49fa-b4ba-2fd44eb2c2f8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:57.176265 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7910af9-140e-49fa-b4ba-2fd44eb2c2f8 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:57.186291 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 397/780] 10.222.0.25 () {56 vars in 1087 bytes} [Fri Oct 28 09:31:56 2022] GET /identity/v3/services => generated 772 bytes in 310 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:31:57.305253 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db2a066-43a9-434c-b548-3ae0854395ab None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:57.305716 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db2a066-43a9-434c-b548-3ae0854395ab None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:57.306095 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db2a066-43a9-434c-b548-3ae0854395ab None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:57.307854 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 384/781] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:57.313593 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5219377-4228-437d-aff6-84c82fba0a26 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:57.314117 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5219377-4228-437d-aff6-84c82fba0a26 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:57.314348 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5219377-4228-437d-aff6-84c82fba0a26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:57.322468 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:57.336394 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-c5219377-4228-437d-aff6-84c82fba0a26 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:57.337578 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-c5219377-4228-437d-aff6-84c82fba0a26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:57.417167 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 398/782] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:57 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 09:31:57.741504 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db2a066-43a9-434c-b548-3ae0854395ab None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:57.759307 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ecd3879-1056-4012-afcd-ea53b195239a None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:57.759651 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ecd3879-1056-4012-afcd-ea53b195239a None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:57.759879 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ecd3879-1056-4012-afcd-ea53b195239a None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:57.761506 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 399/783] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:57.770550 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c8563f-27b1-458e-9bd6-a808a6a651fb None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:57.770851 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c8563f-27b1-458e-9bd6-a808a6a651fb None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:57.771115 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c8563f-27b1-458e-9bd6-a808a6a651fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:57.790431 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:57.807460 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-03c8563f-27b1-458e-9bd6-a808a6a651fb None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:57.808687 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-03c8563f-27b1-458e-9bd6-a808a6a651fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:57.909719 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db2a066-43a9-434c-b548-3ae0854395ab None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:57.938283 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 400/784] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:57 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 09:31:57.971180 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7613add7-86a5-4e1f-b84a-7f6c67d69f75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:57.971829 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7613add7-86a5-4e1f-b84a-7f6c67d69f75 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:57.972111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7613add7-86a5-4e1f-b84a-7f6c67d69f75 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:57.972387 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7613add7-86a5-4e1f-b84a-7f6c67d69f75 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:57.989175 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7613add7-86a5-4e1f-b84a-7f6c67d69f75 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:31:58.002806 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-7613add7-86a5-4e1f-b84a-7f6c67d69f75 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:31:58.002806 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:58.002806 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:58.002806 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:58.002806 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:58.002806 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:58.002806 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:58.002806 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:58.002806 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:58.002806 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:58.002806 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:58.002806 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:58.002806 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:31:58.002806 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:31:58.002806 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:31:58.002806 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:31:58.003695 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:31:58.003695 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:31:58.003695 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:31:58.003695 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:31:58.003695 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:58.003695 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:58.003695 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:58.003695 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:58.003695 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:58.003695 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:58.003695 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:58.003695 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:58.003695 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:58.003695 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:58.003695 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:58.003695 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:58.004516 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:58.004516 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:58.004516 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:58.004516 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:58.004516 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:31:58.004516 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:31:58.004516 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:31:58.004516 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:31:58.004516 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:31:58.004516 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:31:58.004516 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:31:58.004516 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:58.004516 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 385/785] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Oct 28 09:31:57 2022] GET /identity/v3/roles/admin => generated 83 bytes in 262 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:58.008855 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c8563f-27b1-458e-9bd6-a808a6a651fb None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:58.089283 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:58.138153 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c8563f-27b1-458e-9bd6-a808a6a651fb None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:58.174601 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b288e4e7-8286-4175-8a52-047fbba7bd19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:58.175539 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b288e4e7-8286-4175-8a52-047fbba7bd19 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:58.175969 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b288e4e7-8286-4175-8a52-047fbba7bd19 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:58.176396 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b288e4e7-8286-4175-8a52-047fbba7bd19 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:58.177462 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b288e4e7-8286-4175-8a52-047fbba7bd19 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:58.177931 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b288e4e7-8286-4175-8a52-047fbba7bd19 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:58.180362 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b288e4e7-8286-4175-8a52-047fbba7bd19 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:58.193117 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 401/786] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:31:58 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:58.199376 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7613add7-86a5-4e1f-b84a-7f6c67d69f75 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:58.356089 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:58.364461 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7613add7-86a5-4e1f-b84a-7f6c67d69f75 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:58.396563 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b288e4e7-8286-4175-8a52-047fbba7bd19 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:58.432525 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6b12c3-1c65-4f1b-93b0-7fa540ced8be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:58.434050 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6b12c3-1c65-4f1b-93b0-7fa540ced8be None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:58.434856 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6b12c3-1c65-4f1b-93b0-7fa540ced8be None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:58.435629 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6b12c3-1c65-4f1b-93b0-7fa540ced8be None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:58.472390 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c6b12c3-1c65-4f1b-93b0-7fa540ced8be None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:58.476192 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c6b12c3-1c65-4f1b-93b0-7fa540ced8be None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:58.513432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-8c6b12c3-1c65-4f1b-93b0-7fa540ced8be None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:58.513432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:58.513432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:58.513432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:58.513432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:58.513432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:58.513432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:58.513432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:58.513432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:58.513432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:58.513432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:58.513432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:58.513432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:58.513432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:58.513432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:58.513432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:58.515304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:58.515304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:58.515304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:58.515304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:58.515304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:58.515304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:58.515304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:58.515304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:58.515304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:58.515304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:58.515304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:58.515304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:58.515304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:58.515304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:58.515304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:58.515304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:58.515304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:58.517917 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:58.517917 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:58.517917 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:58.517917 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 386/787] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 317 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:58.517917 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57728558-fb18-4d62-9e16-190bc087f2ef None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:58.517917 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57728558-fb18-4d62-9e16-190bc087f2ef None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:58.517917 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57728558-fb18-4d62-9e16-190bc087f2ef None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:58.521603 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 387/788] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:58 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:58.527753 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57728558-fb18-4d62-9e16-190bc087f2ef None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:58.580502 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b288e4e7-8286-4175-8a52-047fbba7bd19 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:58.613500 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:58.626354 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc40ffa9-7708-4606-956f-ad50d97b9b5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:58.627424 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc40ffa9-7708-4606-956f-ad50d97b9b5b None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:58.627921 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc40ffa9-7708-4606-956f-ad50d97b9b5b None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:58.628386 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc40ffa9-7708-4606-956f-ad50d97b9b5b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:58.642507 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc40ffa9-7708-4606-956f-ad50d97b9b5b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:31:58.649495 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-cc40ffa9-7708-4606-956f-ad50d97b9b5b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:31:58.649495 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:58.649495 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:58.649495 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:58.649495 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:58.649495 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:58.649495 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:58.649495 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:58.649495 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:58.649495 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:58.649495 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:58.649495 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:58.649495 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:31:58.649495 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:31:58.649495 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:31:58.649495 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:31:58.650575 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:31:58.650575 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:31:58.650575 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:31:58.650575 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:31:58.650575 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:58.650575 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:58.650575 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:58.650575 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:58.650575 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:58.650575 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:58.650575 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:58.650575 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:58.650575 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:58.650575 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:58.650575 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:58.650575 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:58.651434 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:58.651434 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:58.651434 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:58.651434 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:58.651434 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:31:58.651434 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:31:58.651434 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:31:58.651434 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:31:58.651434 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:31:58.651434 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:31:58.651434 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:31:58.651434 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:58.658775 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 402/789] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Oct 28 09:31:58 2022] GET /identity/v3/roles/admin => generated 83 bytes in 258 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:58.660825 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f78b7a-61e3-4ac4-9883-44f937a4eae9 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:58.661109 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f78b7a-61e3-4ac4-9883-44f937a4eae9 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:58.661412 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98f78b7a-61e3-4ac4-9883-44f937a4eae9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:58.673286 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57728558-fb18-4d62-9e16-190bc087f2ef None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:58.682040 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-98f78b7a-61e3-4ac4-9883-44f937a4eae9 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:31:58.683282 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-98f78b7a-61e3-4ac4-9883-44f937a4eae9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:31:58.715943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b042a1f9-b77b-4f00-9346-dd3076fae570 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:58.717054 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b042a1f9-b77b-4f00-9346-dd3076fae570 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:58.717541 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b042a1f9-b77b-4f00-9346-dd3076fae570 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:58.718028 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b042a1f9-b77b-4f00-9346-dd3076fae570 None admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:58.719040 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b042a1f9-b77b-4f00-9346-dd3076fae570 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:58.719619 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b042a1f9-b77b-4f00-9346-dd3076fae570 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:58.721182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b042a1f9-b77b-4f00-9346-dd3076fae570 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:58.734501 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 388/790] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:58.735605 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b042a1f9-b77b-4f00-9346-dd3076fae570 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:58.754794 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:58.789117 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 403/791] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:31:58 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:31:58.791243 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f78b7a-61e3-4ac4-9883-44f937a4eae9 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:58.843246 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b042a1f9-b77b-4f00-9346-dd3076fae570 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:58.885605 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8422e3fd-4ca2-492a-a973-66604d85542d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:58.886575 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8422e3fd-4ca2-492a-a973-66604d85542d None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:58.887099 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8422e3fd-4ca2-492a-a973-66604d85542d None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:58.887552 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8422e3fd-4ca2-492a-a973-66604d85542d None admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:58.888982 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8422e3fd-4ca2-492a-a973-66604d85542d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:58.889480 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8422e3fd-4ca2-492a-a973-66604d85542d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:58.891368 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8422e3fd-4ca2-492a-a973-66604d85542d None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:58.903350 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 389/792] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:31:58 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:58.909506 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8422e3fd-4ca2-492a-a973-66604d85542d None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:58.948385 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98f78b7a-61e3-4ac4-9883-44f937a4eae9 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:58.980388 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:59.020120 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6180616-84a8-4832-9f24-b711a89d9d4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:59.021224 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6180616-84a8-4832-9f24-b711a89d9d4f None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:59.022919 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6180616-84a8-4832-9f24-b711a89d9d4f None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:59.023855 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6180616-84a8-4832-9f24-b711a89d9d4f None admin] PATH_INFO: `/v3/users/swift` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:59.038706 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:59.040134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6180616-84a8-4832-9f24-b711a89d9d4f None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:59.043900 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6180616-84a8-4832-9f24-b711a89d9d4f None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:59.063847 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-b6180616-84a8-4832-9f24-b711a89d9d4f None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 28 09:31:59.063847 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:59.063847 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:59.063847 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:59.063847 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:59.063847 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:59.063847 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:59.063847 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:59.063847 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:59.063847 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:59.063847 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:59.063847 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:59.063847 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 09:31:59.063847 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 09:31:59.063847 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 09:31:59.063847 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 09:31:59.065039 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:59.065039 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:59.065039 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 09:31:59.065039 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:31:59.065039 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 09:31:59.065039 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:31:59.065039 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:59.065039 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:59.065039 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:59.065039 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:59.065039 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:59.065039 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:59.065039 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:59.065039 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:59.065039 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:59.065039 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:59.065039 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:59.065039 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:59.066171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:59.066171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:59.066171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 09:31:59.066171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 09:31:59.066171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 09:31:59.066171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 09:31:59.066171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 09:31:59.066171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 09:31:59.066171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 28 09:31:59.066171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:31:59.073044 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 404/793] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Oct 28 09:31:58 2022] GET /identity/v3/users/swift => generated 83 bytes in 279 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:59.074502 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6180616-84a8-4832-9f24-b711a89d9d4f None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:59.112505 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8422e3fd-4ca2-492a-a973-66604d85542d None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:59.171323 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0166d1e-96bb-4dc2-9b4e-d2a6920b2423 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:59.171998 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0166d1e-96bb-4dc2-9b4e-d2a6920b2423 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:59.172370 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0166d1e-96bb-4dc2-9b4e-d2a6920b2423 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:59.172663 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0166d1e-96bb-4dc2-9b4e-d2a6920b2423 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:59.185624 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0166d1e-96bb-4dc2-9b4e-d2a6920b2423 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:59.187125 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0166d1e-96bb-4dc2-9b4e-d2a6920b2423 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:59.203069 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-c0166d1e-96bb-4dc2-9b4e-d2a6920b2423 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:59.203069 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:59.203069 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:59.203069 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:59.203069 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:59.203069 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:59.203069 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:59.203069 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:59.203069 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:59.203069 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:59.203069 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:59.203069 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:59.203069 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:59.203069 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:59.203069 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:59.203069 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:59.204074 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:59.204074 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:59.204074 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:59.204074 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:59.204074 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:59.204074 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:59.204074 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:59.204074 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:59.204074 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:59.204074 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:59.204074 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:59.204074 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:59.204074 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:59.204074 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:59.204074 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:59.204074 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:59.204074 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:59.205673 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:59.205673 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:59.205673 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:59.205673 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 390/794] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 295 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:59.205673 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0166d1e-96bb-4dc2-9b4e-d2a6920b2423 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:59.221264 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6180616-84a8-4832-9f24-b711a89d9d4f None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:59.262322 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37344d40-8920-41f0-9c26-6d325ac169e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:59.262973 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37344d40-8920-41f0-9c26-6d325ac169e9 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:59.263259 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37344d40-8920-41f0-9c26-6d325ac169e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:59.263534 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37344d40-8920-41f0-9c26-6d325ac169e9 None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:59.264270 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37344d40-8920-41f0-9c26-6d325ac169e9 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:59.264580 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37344d40-8920-41f0-9c26-6d325ac169e9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:59.266932 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37344d40-8920-41f0-9c26-6d325ac169e9 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:59.290523 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 405/795] 10.222.0.25 () {56 vars in 1135 bytes} [Fri Oct 28 09:31:59 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:59.291456 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37344d40-8920-41f0-9c26-6d325ac169e9 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:59.328665 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0166d1e-96bb-4dc2-9b4e-d2a6920b2423 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:59.329568 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:59.352392 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:59.369909 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d98cd6-a2cc-4422-94d9-7045de21fc9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:59.371609 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d98cd6-a2cc-4422-94d9-7045de21fc9b None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:59.372495 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d98cd6-a2cc-4422-94d9-7045de21fc9b None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:59.372974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d98cd6-a2cc-4422-94d9-7045de21fc9b None admin] PATH_INFO: `/v3/roles/service` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:59.389200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48d98cd6-a2cc-4422-94d9-7045de21fc9b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:31:59.396067 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-48d98cd6-a2cc-4422-94d9-7045de21fc9b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 09:31:59.396067 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:59.396067 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:59.396067 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:59.396067 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:59.396067 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:59.396067 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:59.396067 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:59.396067 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:59.396067 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:59.396067 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:59.396067 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:59.396067 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:31:59.396067 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:31:59.396067 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:31:59.396067 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:31:59.397005 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:31:59.397005 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:31:59.397005 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:31:59.397005 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:31:59.397005 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:59.397005 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:59.397005 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:59.397005 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:59.397005 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:59.397005 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:59.397005 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:59.397005 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:59.397005 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:59.397005 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:59.397005 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:59.397005 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:59.397917 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:59.397917 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:59.397917 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:59.397917 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:59.397917 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:31:59.397917 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:31:59.397917 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:31:59.397917 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:31:59.397917 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:31:59.397917 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:31:59.397917 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 09:31:59.397917 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:59.404918 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 391/796] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Oct 28 09:31:59 2022] GET /identity/v3/roles/service => generated 85 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:59.404918 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d98cd6-a2cc-4422-94d9-7045de21fc9b None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:59.419814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37344d40-8920-41f0-9c26-6d325ac169e9 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:59.465227 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0283794-95ac-432f-99ce-b8222de3b240 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:59.465917 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0283794-95ac-432f-99ce-b8222de3b240 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:59.466299 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0283794-95ac-432f-99ce-b8222de3b240 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:59.466878 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0283794-95ac-432f-99ce-b8222de3b240 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:59.467932 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0283794-95ac-432f-99ce-b8222de3b240 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:59.468356 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0283794-95ac-432f-99ce-b8222de3b240 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:59.470282 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0283794-95ac-432f-99ce-b8222de3b240 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:59.486749 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 406/797] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:59.488148 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0283794-95ac-432f-99ce-b8222de3b240 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:59.513030 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d98cd6-a2cc-4422-94d9-7045de21fc9b None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:59.549640 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-358e952f-6787-4eb7-aedc-c8004cec6b08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:59.550701 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358e952f-6787-4eb7-aedc-c8004cec6b08 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:59.551289 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358e952f-6787-4eb7-aedc-c8004cec6b08 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:59.551780 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-358e952f-6787-4eb7-aedc-c8004cec6b08 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:59.582748 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-358e952f-6787-4eb7-aedc-c8004cec6b08 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:59.584281 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-358e952f-6787-4eb7-aedc-c8004cec6b08 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:59.605255 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-358e952f-6787-4eb7-aedc-c8004cec6b08 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:59.605255 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:59.605255 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:59.605255 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:59.605255 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:59.605255 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:59.605255 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:59.605255 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:59.605255 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:59.605255 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:59.605255 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:59.605255 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:59.605255 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:31:59.605255 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:31:59.605255 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:31:59.605255 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:31:59.606263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:59.606263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:59.606263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:59.606263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:59.606263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:59.606263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:59.606263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:59.606263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:59.606263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:59.606263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:59.606263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:59.606263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:59.606263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:59.606263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:59.606263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:59.606263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:59.606263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:31:59.607211 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:31:59.607211 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:31:59.607211 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:59.614600 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 392/798] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:59.614600 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-358e952f-6787-4eb7-aedc-c8004cec6b08 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:59.621075 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:59.630290 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0283794-95ac-432f-99ce-b8222de3b240 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:59.656948 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:59.678943 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f778a5c-5713-4618-8ce3-e5d055000848 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:59.680227 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f778a5c-5713-4618-8ce3-e5d055000848 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:59.680513 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f778a5c-5713-4618-8ce3-e5d055000848 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:59.680832 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f778a5c-5713-4618-8ce3-e5d055000848 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:59.681482 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f778a5c-5713-4618-8ce3-e5d055000848 None admin] RBAC: Adding query filter params (name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:59.681790 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f778a5c-5713-4618-8ce3-e5d055000848 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:59.683206 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f778a5c-5713-4618-8ce3-e5d055000848 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:59.692622 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 407/799] 10.222.0.25 () {56 vars in 1103 bytes} [Fri Oct 28 09:31:59 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:59.694470 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f778a5c-5713-4618-8ce3-e5d055000848 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:59.740016 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-358e952f-6787-4eb7-aedc-c8004cec6b08 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:59.776959 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf1ac724-c87e-4933-a1e8-b70bf2cb58e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:59.777658 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1ac724-c87e-4933-a1e8-b70bf2cb58e2 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:59.777875 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1ac724-c87e-4933-a1e8-b70bf2cb58e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:59.778150 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf1ac724-c87e-4933-a1e8-b70bf2cb58e2 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:59.785969 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf1ac724-c87e-4933-a1e8-b70bf2cb58e2 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:59.787464 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf1ac724-c87e-4933-a1e8-b70bf2cb58e2 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:59.799676 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-cf1ac724-c87e-4933-a1e8-b70bf2cb58e2 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 28 09:31:59.799676 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:31:59.799676 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:31:59.799676 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:31:59.799676 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:31:59.799676 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:31:59.799676 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:31:59.799676 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:31:59.799676 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:31:59.799676 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:31:59.799676 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:31:59.799676 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:31:59.799676 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 09:31:59.799676 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 09:31:59.799676 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 09:31:59.799676 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 09:31:59.800682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:31:59.800682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:31:59.800682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 09:31:59.800682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:31:59.800682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 09:31:59.800682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:31:59.800682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:31:59.800682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:31:59.800682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:31:59.800682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:31:59.800682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:31:59.800682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:31:59.800682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:31:59.800682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:31:59.800682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:31:59.800682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:31:59.800682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:31:59.800682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:31:59.807602 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:31:59.807602 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:31:59.807602 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 09:31:59.807602 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 09:31:59.807602 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 09:31:59.807602 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 09:31:59.807602 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 09:31:59.807602 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 09:31:59.807602 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 28 09:31:59.807602 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:31:59.808140 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f778a5c-5713-4618-8ce3-e5d055000848 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:59.808215 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 393/800] 10.222.0.25 () {56 vars in 1093 bytes} [Fri Oct 28 09:31:59 2022] GET /identity/v3/users/nova => generated 82 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:31:59.808215 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b010ad86-2dbe-49b9-9441-73077691da9a None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:59.808215 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b010ad86-2dbe-49b9-9441-73077691da9a None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:59.808215 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b010ad86-2dbe-49b9-9441-73077691da9a None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:59.808215 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 394/801] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:31:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:31:59.808215 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b010ad86-2dbe-49b9-9441-73077691da9a None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:59.843377 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0d141a-9c28-47a4-be16-43f9b7d5d188 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:59.843982 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0d141a-9c28-47a4-be16-43f9b7d5d188 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:59.844258 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0d141a-9c28-47a4-be16-43f9b7d5d188 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:59.844528 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0d141a-9c28-47a4-be16-43f9b7d5d188 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:59.845186 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be0d141a-9c28-47a4-be16-43f9b7d5d188 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:31:59.845459 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be0d141a-9c28-47a4-be16-43f9b7d5d188 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:59.846828 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be0d141a-9c28-47a4-be16-43f9b7d5d188 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:31:59.858822 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 408/802] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:31:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:31:59.859659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0d141a-9c28-47a4-be16-43f9b7d5d188 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:31:59.881667 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:59.926040 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:31:59.937444 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b010ad86-2dbe-49b9-9441-73077691da9a None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:59.974459 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3ceedf4-799e-4e2c-a89d-f984b16f4b1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:31:59.975112 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ceedf4-799e-4e2c-a89d-f984b16f4b1d None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:31:59.975458 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ceedf4-799e-4e2c-a89d-f984b16f4b1d None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:31:59.975736 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ceedf4-799e-4e2c-a89d-f984b16f4b1d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:31:59.975836 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0d141a-9c28-47a4-be16-43f9b7d5d188 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:31:59.987570 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3ceedf4-799e-4e2c-a89d-f984b16f4b1d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:31:59.989041 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3ceedf4-799e-4e2c-a89d-f984b16f4b1d None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:00.006943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-b3ceedf4-799e-4e2c-a89d-f984b16f4b1d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:32:00.006943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:00.006943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:00.006943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:00.006943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:00.006943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:00.006943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:00.006943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:00.006943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:00.006943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:00.006943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:00.006943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:00.006943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:32:00.006943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:32:00.006943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:32:00.006943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:32:00.008101 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:00.008101 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:00.008101 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:00.008101 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:00.008101 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:00.008101 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:00.008101 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:00.008101 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:00.008101 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:00.008101 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:00.008101 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:00.008101 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:00.008101 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:00.008101 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:00.008101 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:00.008101 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:00.008101 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:32:00.009246 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:32:00.009246 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:32:00.009246 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:00.012559 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 395/803] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:31:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:00.013946 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0d1bda-5dec-44fc-bbb6-8cf1f3eedd78 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:00.014239 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0d1bda-5dec-44fc-bbb6-8cf1f3eedd78 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:00.014546 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0d1bda-5dec-44fc-bbb6-8cf1f3eedd78 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:00.014682 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307251e4-c6db-437d-9ec0-60a158e31c89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:00.015293 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307251e4-c6db-437d-9ec0-60a158e31c89 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:00.015570 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307251e4-c6db-437d-9ec0-60a158e31c89 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:00.015841 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307251e4-c6db-437d-9ec0-60a158e31c89 None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:00.016561 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-307251e4-c6db-437d-9ec0-60a158e31c89 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:00.016862 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-307251e4-c6db-437d-9ec0-60a158e31c89 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:00.018257 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-307251e4-c6db-437d-9ec0-60a158e31c89 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:00.034150 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-be0d1bda-5dec-44fc-bbb6-8cf1f3eedd78 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:32:00.035386 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-be0d1bda-5dec-44fc-bbb6-8cf1f3eedd78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:32:00.037633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 409/804] 10.222.0.25 () {56 vars in 1133 bytes} [Fri Oct 28 09:31:59 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:00.039085 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307251e4-c6db-437d-9ec0-60a158e31c89 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:00.103316 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 396/805] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:32:00 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:32:00.104173 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0d1bda-5dec-44fc-bbb6-8cf1f3eedd78 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:00.116430 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:00.144692 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307251e4-c6db-437d-9ec0-60a158e31c89 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:00.182600 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3478eca9-1b35-47f7-a99e-2f813d3119b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:00.183239 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3478eca9-1b35-47f7-a99e-2f813d3119b8 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:00.183527 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3478eca9-1b35-47f7-a99e-2f813d3119b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:00.183802 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3478eca9-1b35-47f7-a99e-2f813d3119b8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:00.191229 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:00.191555 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3478eca9-1b35-47f7-a99e-2f813d3119b8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:00.192991 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3478eca9-1b35-47f7-a99e-2f813d3119b8 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:00.212838 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-3478eca9-1b35-47f7-a99e-2f813d3119b8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:32:00.212838 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:00.212838 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:00.212838 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:00.212838 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:00.212838 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:00.212838 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:00.212838 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:00.212838 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:00.212838 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:00.212838 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:00.212838 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:00.212838 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 09:32:00.212838 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 09:32:00.212838 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 09:32:00.213798 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 09:32:00.213798 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:00.213798 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:00.213798 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:00.213798 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:00.213798 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:00.213798 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:00.213798 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:00.213798 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:00.213798 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:00.213798 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:00.213798 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:00.213798 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:00.213798 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:00.213798 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:00.213798 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:00.213798 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:00.214751 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 09:32:00.214751 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 09:32:00.214751 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 09:32:00.214751 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 09:32:00.214751 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 09:32:00.214751 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 09:32:00.214751 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:32:00.214751 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:00.214751 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 410/806] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:32:00 2022] GET /identity/v3/projects/service => generated 88 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:00.215838 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334ed1d4-1f8b-4f69-994e-39df1eb0d054 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:00.215838 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334ed1d4-1f8b-4f69-994e-39df1eb0d054 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:00.216085 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334ed1d4-1f8b-4f69-994e-39df1eb0d054 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:00.217416 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 411/807] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:32:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:32:00.218025 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-334ed1d4-1f8b-4f69-994e-39df1eb0d054 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:00.228005 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0d1bda-5dec-44fc-bbb6-8cf1f3eedd78 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:00.265014 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-209063b7-ba49-4abe-9bda-76d1f4e0625f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:00.265692 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209063b7-ba49-4abe-9bda-76d1f4e0625f None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:00.266073 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209063b7-ba49-4abe-9bda-76d1f4e0625f None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:00.266488 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209063b7-ba49-4abe-9bda-76d1f4e0625f None admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:00.267239 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-209063b7-ba49-4abe-9bda-76d1f4e0625f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:00.267774 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-209063b7-ba49-4abe-9bda-76d1f4e0625f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:00.269213 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-209063b7-ba49-4abe-9bda-76d1f4e0625f None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:00.285915 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 397/808] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:32:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:00.285915 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-209063b7-ba49-4abe-9bda-76d1f4e0625f None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:00.335683 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-334ed1d4-1f8b-4f69-994e-39df1eb0d054 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:00.372695 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f773813-8290-4256-9351-d4d4ef33d9f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:00.373341 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f773813-8290-4256-9351-d4d4ef33d9f6 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:00.373620 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f773813-8290-4256-9351-d4d4ef33d9f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:00.373890 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f773813-8290-4256-9351-d4d4ef33d9f6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:00.376654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:00.384186 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-209063b7-ba49-4abe-9bda-76d1f4e0625f None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:00.417182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e433a83-1d8b-40e3-9f86-18b0eeb1b853 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:00.417789 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e433a83-1d8b-40e3-9f86-18b0eeb1b853 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:00.418063 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e433a83-1d8b-40e3-9f86-18b0eeb1b853 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:00.418332 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e433a83-1d8b-40e3-9f86-18b0eeb1b853 None admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:00.419053 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e433a83-1d8b-40e3-9f86-18b0eeb1b853 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:00.419358 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e433a83-1d8b-40e3-9f86-18b0eeb1b853 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:00.420776 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e433a83-1d8b-40e3-9f86-18b0eeb1b853 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:00.422342 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:00.434221 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 398/809] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Oct 28 09:32:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:00.435468 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22359009-8797-49f0-a8e3-71c5db76df98 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:00.435749 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22359009-8797-49f0-a8e3-71c5db76df98 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:00.436020 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22359009-8797-49f0-a8e3-71c5db76df98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:00.449472 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f773813-8290-4256-9351-d4d4ef33d9f6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:00.450955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f773813-8290-4256-9351-d4d4ef33d9f6 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:00.453504 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-22359009-8797-49f0-a8e3-71c5db76df98 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:32:00.454718 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-22359009-8797-49f0-a8e3-71c5db76df98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:32:00.521317 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 399/810] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:32:00 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:32:00.522640 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22359009-8797-49f0-a8e3-71c5db76df98 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:00.554722 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 412/811] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:32:00 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 332 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 09:32:00.554722 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f773813-8290-4256-9351-d4d4ef33d9f6 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:00.590260 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:00.658987 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22359009-8797-49f0-a8e3-71c5db76df98 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:00.666613 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f773813-8290-4256-9351-d4d4ef33d9f6 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:00.684218 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:00.696896 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93347da0-2d1c-4091-ad14-7a51c88a9718 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:00.697541 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93347da0-2d1c-4091-ad14-7a51c88a9718 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:00.697827 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93347da0-2d1c-4091-ad14-7a51c88a9718 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:00.698112 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93347da0-2d1c-4091-ad14-7a51c88a9718 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:00.705555 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e775d9b2-5f48-4d11-bfc4-1e0c205a7a6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:00.705714 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93347da0-2d1c-4091-ad14-7a51c88a9718 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:00.706188 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e775d9b2-5f48-4d11-bfc4-1e0c205a7a6f None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:00.706501 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e775d9b2-5f48-4d11-bfc4-1e0c205a7a6f None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:00.706780 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e775d9b2-5f48-4d11-bfc4-1e0c205a7a6f None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:00.707039 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93347da0-2d1c-4091-ad14-7a51c88a9718 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:00.707368 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e775d9b2-5f48-4d11-bfc4-1e0c205a7a6f None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:00.708719 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e775d9b2-5f48-4d11-bfc4-1e0c205a7a6f None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:00.717688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-e775d9b2-5f48-4d11-bfc4-1e0c205a7a6f None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 28 09:32:00.717688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:00.717688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 09:32:00.717688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 09:32:00.717688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 09:32:00.717688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 09:32:00.717688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 09:32:00.717688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:32:00.717688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 28 09:32:00.717688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:00.717688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 09:32:00.717688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:00.717688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:00.717688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:00.717688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:00.717688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:00.717688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:00.718754 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:00.718754 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:00.718754 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:00.718754 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:00.718754 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:00.718754 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:00.718754 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 09:32:00.718754 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 09:32:00.718754 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 09:32:00.718754 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 09:32:00.718754 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:00.718754 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:00.718754 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:00.718754 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:00.718754 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:00.718754 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:00.719619 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:00.719619 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:00.719619 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:00.719619 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:00.719619 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:00.719619 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:00.719619 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:00.719619 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:00.719619 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:00.719619 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:00.719619 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 09:32:00.719619 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:32:00.719619 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Oct 28 09:32:00.719619 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:00.719619 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 413/812] 10.222.0.25 () {56 vars in 1129 bytes} [Fri Oct 28 09:32:00 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:00.720475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-93347da0-2d1c-4091-ad14-7a51c88a9718 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 28 09:32:00.720475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:00.720475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:00.720475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:00.720475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:00.720475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:00.720475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:00.720475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:00.720475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:00.720475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:00.720475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:00.720475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:00.720475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 09:32:00.720475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 09:32:00.720475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 09:32:00.720475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 09:32:00.721337 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:00.721337 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:00.721337 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 09:32:00.721337 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:32:00.721337 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 09:32:00.721337 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:32:00.721337 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:00.721337 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:00.721337 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:00.721337 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:00.721337 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:00.721337 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:00.721337 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:00.721337 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:00.721337 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:00.721337 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:00.721337 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:00.721337 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:00.723808 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:00.723808 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:00.723808 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 09:32:00.723808 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 09:32:00.723808 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 09:32:00.723808 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 09:32:00.723808 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 09:32:00.723808 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 09:32:00.723808 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 28 09:32:00.723808 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:00.725031 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e775d9b2-5f48-4d11-bfc4-1e0c205a7a6f None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:00.725238 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 400/813] 10.222.0.25 () {56 vars in 1117 bytes} [Fri Oct 28 09:32:00 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:00.725238 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93347da0-2d1c-4091-ad14-7a51c88a9718 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:00.829705 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:00.833628 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e775d9b2-5f48-4d11-bfc4-1e0c205a7a6f None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:00.834524 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93347da0-2d1c-4091-ad14-7a51c88a9718 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:00.869281 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9216d8b-35d5-4626-b734-df0f7e775220 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:00.869508 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8851a364-75d3-4d0d-a1c9-bfe3ecde0620 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:00.869948 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9216d8b-35d5-4626-b734-df0f7e775220 None admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:00.870053 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8851a364-75d3-4d0d-a1c9-bfe3ecde0620 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:00.870231 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9216d8b-35d5-4626-b734-df0f7e775220 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:00.870325 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8851a364-75d3-4d0d-a1c9-bfe3ecde0620 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:00.870540 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9216d8b-35d5-4626-b734-df0f7e775220 None admin] PATH_INFO: `/v3/projects/455d1c826b974bc38497d18a516867c7/users/9c6cffe0d56e45c4b8fb54fbe5df6094/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:00.870635 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8851a364-75d3-4d0d-a1c9-bfe3ecde0620 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:00.872577 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9216d8b-35d5-4626-b734-df0f7e775220 None admin] RBAC: Authorizing `identity:create_grant(project_id=455d1c826b974bc38497d18a516867c7, user_id=9c6cffe0d56e45c4b8fb54fbe5df6094, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:00.874007 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9216d8b-35d5-4626-b734-df0f7e775220 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:00.882445 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8851a364-75d3-4d0d-a1c9-bfe3ecde0620 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:00.883998 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8851a364-75d3-4d0d-a1c9-bfe3ecde0620 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:00.886789 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 414/814] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Oct 28 09:32:00 2022] PUT /identity/v3/projects/455d1c826b974bc38497d18a516867c7/users/9c6cffe0d56e45c4b8fb54fbe5df6094/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:32:00.887604 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9216d8b-35d5-4626-b734-df0f7e775220 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:00.900665 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-8851a364-75d3-4d0d-a1c9-bfe3ecde0620 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:32:00.900665 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:00.900665 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:00.900665 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:00.900665 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:00.900665 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:00.900665 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:00.900665 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:00.900665 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:00.900665 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:00.900665 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:00.900665 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:00.900665 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:32:00.900665 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:32:00.900665 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:32:00.900665 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:32:00.901705 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:00.901705 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:00.901705 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:00.901705 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:00.901705 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:00.901705 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:00.901705 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:00.901705 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:00.901705 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:00.901705 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:00.901705 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:00.901705 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:00.901705 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:00.901705 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:00.901705 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:00.901705 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:00.901705 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:32:00.903366 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:32:00.903366 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:32:00.903366 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:00.903366 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 401/815] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:32:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:00.903366 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8851a364-75d3-4d0d-a1c9-bfe3ecde0620 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:00.944833 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:01.019434 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9216d8b-35d5-4626-b734-df0f7e775220 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:01.029680 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8851a364-75d3-4d0d-a1c9-bfe3ecde0620 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:01.063188 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d11390-26e9-4d4d-b927-0f305905ecc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:01.063930 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d11390-26e9-4d4d-b927-0f305905ecc4 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:01.064296 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d11390-26e9-4d4d-b927-0f305905ecc4 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:01.064649 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d11390-26e9-4d4d-b927-0f305905ecc4 None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:01.065363 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d11390-26e9-4d4d-b927-0f305905ecc4 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:01.065740 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d11390-26e9-4d4d-b927-0f305905ecc4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:01.067221 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d11390-26e9-4d4d-b927-0f305905ecc4 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:01.073766 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6895ca9-a671-4ee7-93ee-07f612eab86a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:01.074563 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6895ca9-a671-4ee7-93ee-07f612eab86a None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:01.075680 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6895ca9-a671-4ee7-93ee-07f612eab86a None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:01.075680 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6895ca9-a671-4ee7-93ee-07f612eab86a None admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:01.076105 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6895ca9-a671-4ee7-93ee-07f612eab86a None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:01.076503 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6895ca9-a671-4ee7-93ee-07f612eab86a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:01.076947 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 415/816] 10.222.0.25 () {56 vars in 1124 bytes} [Fri Oct 28 09:32:00 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:01.077893 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d11390-26e9-4d4d-b927-0f305905ecc4 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:01.078203 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6895ca9-a671-4ee7-93ee-07f612eab86a None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:01.099392 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 402/817] 10.222.0.25 () {56 vars in 1149 bytes} [Fri Oct 28 09:32:00 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:01.101305 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6895ca9-a671-4ee7-93ee-07f612eab86a None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:01.112186 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:01.197079 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d11390-26e9-4d4d-b927-0f305905ecc4 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:01.198696 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:01.205697 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6895ca9-a671-4ee7-93ee-07f612eab86a None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:01.234940 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8935a9-7f8e-4565-8ebd-dae424cf6b4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:01.235674 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8935a9-7f8e-4565-8ebd-dae424cf6b4d None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:01.235957 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8935a9-7f8e-4565-8ebd-dae424cf6b4d None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:01.236233 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc8935a9-7f8e-4565-8ebd-dae424cf6b4d None admin] PATH_INFO: `/v3/services/network` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:01.236814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc8935a9-7f8e-4565-8ebd-dae424cf6b4d None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:01.238157 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc8935a9-7f8e-4565-8ebd-dae424cf6b4d None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:01.240315 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05e0114c-c0e6-4d11-9809-11d54e28f592 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:01.241454 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e0114c-c0e6-4d11-9809-11d54e28f592 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:01.241741 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e0114c-c0e6-4d11-9809-11d54e28f592 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:01.242015 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e0114c-c0e6-4d11-9809-11d54e28f592 None admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:01.242699 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05e0114c-c0e6-4d11-9809-11d54e28f592 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:01.243009 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05e0114c-c0e6-4d11-9809-11d54e28f592 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:01.244382 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05e0114c-c0e6-4d11-9809-11d54e28f592 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:01.247973 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-dc8935a9-7f8e-4565-8ebd-dae424cf6b4d None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Oct 28 09:32:01.247973 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:01.247973 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 09:32:01.247973 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 09:32:01.247973 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 09:32:01.247973 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 09:32:01.247973 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 09:32:01.247973 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:32:01.247973 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 28 09:32:01.247973 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:01.247973 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 09:32:01.247973 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:01.247973 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:01.247973 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:01.247973 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:01.247973 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:01.247973 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:01.248968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:01.248968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:01.248968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:01.248968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:01.248968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:01.248968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:01.248968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 09:32:01.248968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 09:32:01.248968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 09:32:01.248968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 09:32:01.248968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:01.248968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:01.248968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:01.248968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:01.248968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:01.248968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:01.249814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:01.249814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:01.249814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:01.249814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:01.249814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:01.249814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:01.249814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:01.249814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:01.249814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:01.249814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:01.249814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 09:32:01.249814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:32:01.249814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Oct 28 09:32:01.249814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:01.253994 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 416/818] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:32:01 2022] GET /identity/v3/services/network => generated 88 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:01.254998 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8935a9-7f8e-4565-8ebd-dae424cf6b4d None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:01.261032 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 403/819] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:32:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:01.261032 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05e0114c-c0e6-4d11-9809-11d54e28f592 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:01.350590 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:01.365609 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05e0114c-c0e6-4d11-9809-11d54e28f592 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:01.368054 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc8935a9-7f8e-4565-8ebd-dae424cf6b4d None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:01.401733 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc666aa-5e7e-4a5d-90ca-50fe68d262ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:01.402373 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc666aa-5e7e-4a5d-90ca-50fe68d262ed None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:01.402819 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc666aa-5e7e-4a5d-90ca-50fe68d262ed None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:01.403102 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc666aa-5e7e-4a5d-90ca-50fe68d262ed None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:01.405948 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280bcf35-f73e-4324-8663-e2a230d089a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:01.406848 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280bcf35-f73e-4324-8663-e2a230d089a5 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:01.407274 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280bcf35-f73e-4324-8663-e2a230d089a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:01.407711 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280bcf35-f73e-4324-8663-e2a230d089a5 None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:01.408749 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-280bcf35-f73e-4324-8663-e2a230d089a5 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:01.409210 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-280bcf35-f73e-4324-8663-e2a230d089a5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:01.410866 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-280bcf35-f73e-4324-8663-e2a230d089a5 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:01.414500 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc666aa-5e7e-4a5d-90ca-50fe68d262ed None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:01.415951 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc666aa-5e7e-4a5d-90ca-50fe68d262ed None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:01.421299 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 417/820] 10.222.0.25 () {56 vars in 1124 bytes} [Fri Oct 28 09:32:01 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:01.422330 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280bcf35-f73e-4324-8663-e2a230d089a5 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:01.433675 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-acc666aa-5e7e-4a5d-90ca-50fe68d262ed None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:32:01.433675 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:01.433675 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:01.433675 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:01.433675 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:01.433675 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:01.433675 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:01.433675 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:01.433675 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:01.433675 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:01.433675 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:01.433675 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:01.433675 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:32:01.433675 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:32:01.433675 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:32:01.433675 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:32:01.434694 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:01.434694 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:01.434694 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:01.434694 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:01.434694 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:01.434694 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:01.434694 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:01.434694 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:01.434694 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:01.434694 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:01.434694 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:01.434694 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:01.434694 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:01.434694 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:01.434694 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:01.434694 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:01.434694 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:32:01.435567 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:32:01.435567 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:32:01.435567 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:01.436379 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 404/821] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:32:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:01.437257 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc666aa-5e7e-4a5d-90ca-50fe68d262ed None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:01.447432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:01.537625 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280bcf35-f73e-4324-8663-e2a230d089a5 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:01.544306 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc666aa-5e7e-4a5d-90ca-50fe68d262ed None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:01.585419 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3abd6a69-3cd8-438f-ba26-7454dcf3ce5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:01.586167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abd6a69-3cd8-438f-ba26-7454dcf3ce5e None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:01.586521 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abd6a69-3cd8-438f-ba26-7454dcf3ce5e None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:01.586869 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abd6a69-3cd8-438f-ba26-7454dcf3ce5e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:01.588314 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3619cf-0376-445f-b68f-c4b94a525e6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:01.588997 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3619cf-0376-445f-b68f-c4b94a525e6d None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:01.589348 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3619cf-0376-445f-b68f-c4b94a525e6d None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:01.589695 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3619cf-0376-445f-b68f-c4b94a525e6d None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:01.590420 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb3619cf-0376-445f-b68f-c4b94a525e6d None admin] RBAC: Adding query filter params (name=network) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:01.590801 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb3619cf-0376-445f-b68f-c4b94a525e6d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:01.592230 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb3619cf-0376-445f-b68f-c4b94a525e6d None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:01.594512 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3abd6a69-3cd8-438f-ba26-7454dcf3ce5e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:01.596031 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3abd6a69-3cd8-438f-ba26-7454dcf3ce5e None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:01.597331 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 418/822] 10.222.0.25 () {56 vars in 1112 bytes} [Fri Oct 28 09:32:01 2022] GET /identity/v3/services?name=network => generated 124 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:01.602644 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3619cf-0376-445f-b68f-c4b94a525e6d None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:01.608531 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-3abd6a69-3cd8-438f-ba26-7454dcf3ce5e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:32:01.608531 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:01.608531 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:01.608531 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:01.608531 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:01.608531 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:01.608531 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:01.608531 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:01.608531 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:01.608531 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:01.608531 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:01.608531 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:01.608531 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 09:32:01.608531 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 09:32:01.608531 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 09:32:01.609475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 09:32:01.609475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:01.609475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:01.609475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:01.609475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:01.609475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:01.609475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:01.609475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:01.609475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:01.609475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:01.609475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:01.609475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:01.609475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:01.609475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:01.609475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:01.609475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:01.609475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:01.615290 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 09:32:01.615290 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 09:32:01.615290 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 09:32:01.615290 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 09:32:01.615290 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 09:32:01.615290 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 09:32:01.615290 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:32:01.615290 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:01.615290 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 405/823] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:32:01 2022] GET /identity/v3/projects/service => generated 88 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:01.615290 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3abd6a69-3cd8-438f-ba26-7454dcf3ce5e None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:01.624578 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:01.682195 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:01.719905 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3abd6a69-3cd8-438f-ba26-7454dcf3ce5e None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:01.723786 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3619cf-0376-445f-b68f-c4b94a525e6d None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:01.753433 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0247f26-aa1f-4ce6-adb0-42f5985ef884 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:01.754046 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0247f26-aa1f-4ce6-adb0-42f5985ef884 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:01.754333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0247f26-aa1f-4ce6-adb0-42f5985ef884 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:01.754647 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0247f26-aa1f-4ce6-adb0-42f5985ef884 None admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:01.755306 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0247f26-aa1f-4ce6-adb0-42f5985ef884 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:01.755616 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0247f26-aa1f-4ce6-adb0-42f5985ef884 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:01.757539 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0247f26-aa1f-4ce6-adb0-42f5985ef884 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:01.761568 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3be8b69-ac01-4b19-93f0-b1c178f21729 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:01.762434 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3be8b69-ac01-4b19-93f0-b1c178f21729 None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:01.762917 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3be8b69-ac01-4b19-93f0-b1c178f21729 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:01.763381 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3be8b69-ac01-4b19-93f0-b1c178f21729 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:01.764231 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3be8b69-ac01-4b19-93f0-b1c178f21729 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:01.765876 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3be8b69-ac01-4b19-93f0-b1c178f21729 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:01.770309 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 406/824] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:32:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:01.771226 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0247f26-aa1f-4ce6-adb0-42f5985ef884 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:01.806423 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 419/825] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:32:01 2022] POST /identity/v3/endpoints => generated 346 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:32:01.806423 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3be8b69-ac01-4b19-93f0-b1c178f21729 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:01.901695 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:01.917312 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0247f26-aa1f-4ce6-adb0-42f5985ef884 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:01.953346 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3be8b69-ac01-4b19-93f0-b1c178f21729 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:01.961980 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:01.967999 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa120cfa-3cf6-42d9-8c97-f5fb4e357e36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:01.969003 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa120cfa-3cf6-42d9-8c97-f5fb4e357e36 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:01.969297 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa120cfa-3cf6-42d9-8c97-f5fb4e357e36 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:01.969579 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa120cfa-3cf6-42d9-8c97-f5fb4e357e36 None admin] PATH_INFO: `/v3/services` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:01.970236 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa120cfa-3cf6-42d9-8c97-f5fb4e357e36 None admin] RBAC: Adding query filter params (type=network) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:01.970893 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa120cfa-3cf6-42d9-8c97-f5fb4e357e36 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:01.972860 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa120cfa-3cf6-42d9-8c97-f5fb4e357e36 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:01.984915 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 407/826] 10.222.0.25 () {56 vars in 1112 bytes} [Fri Oct 28 09:32:01 2022] GET /identity/v3/services?type=network => generated 350 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:01.985691 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa120cfa-3cf6-42d9-8c97-f5fb4e357e36 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:01.999049 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca238024-64e2-4fed-af94-03cfcc5256f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:01.999879 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca238024-64e2-4fed-af94-03cfcc5256f3 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:02.000250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca238024-64e2-4fed-af94-03cfcc5256f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:02.000591 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca238024-64e2-4fed-af94-03cfcc5256f3 None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:02.001302 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca238024-64e2-4fed-af94-03cfcc5256f3 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:02.001611 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca238024-64e2-4fed-af94-03cfcc5256f3 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:02.003184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca238024-64e2-4fed-af94-03cfcc5256f3 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:02.017116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 420/827] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Oct 28 09:32:01 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:02.017773 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca238024-64e2-4fed-af94-03cfcc5256f3 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:02.104663 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa120cfa-3cf6-42d9-8c97-f5fb4e357e36 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:02.146288 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca238024-64e2-4fed-af94-03cfcc5256f3 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:02.151508 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a91627-ee1f-4855-a6c2-2a56b11bfbbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:02.152213 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a91627-ee1f-4855-a6c2-2a56b11bfbbe None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:02.152439 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a91627-ee1f-4855-a6c2-2a56b11bfbbe None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:02.152718 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a91627-ee1f-4855-a6c2-2a56b11bfbbe None admin] PATH_INFO: `/v3/projects/service` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:02.161984 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a91627-ee1f-4855-a6c2-2a56b11bfbbe None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:02.163890 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a91627-ee1f-4855-a6c2-2a56b11bfbbe None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:02.177467 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:02.181961 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-77a91627-ee1f-4855-a6c2-2a56b11bfbbe None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:32:02.181961 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:02.181961 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:02.181961 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:02.181961 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:02.181961 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:02.181961 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:02.181961 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:02.181961 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:02.181961 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:02.181961 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:02.181961 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:02.181961 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 09:32:02.181961 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 09:32:02.181961 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 09:32:02.183058 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 09:32:02.183058 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:02.183058 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:02.183058 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:02.183058 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:02.183058 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:02.183058 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:02.183058 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:02.183058 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:02.183058 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:02.183058 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:02.183058 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:02.183058 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:02.183058 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:02.183058 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:02.183058 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:02.183058 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:02.184736 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 09:32:02.184736 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 09:32:02.184736 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 09:32:02.184736 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 09:32:02.184736 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 09:32:02.184736 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 09:32:02.184736 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:32:02.184736 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:02.184736 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 408/828] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:32:01 2022] GET /identity/v3/projects/service => generated 88 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:02.185229 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a91627-ee1f-4855-a6c2-2a56b11bfbbe None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:02.192016 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ccc2625-7fa8-467d-8d9f-ca0e255f234b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:02.192961 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ccc2625-7fa8-467d-8d9f-ca0e255f234b None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:02.193449 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ccc2625-7fa8-467d-8d9f-ca0e255f234b None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:02.194048 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ccc2625-7fa8-467d-8d9f-ca0e255f234b None admin] PATH_INFO: `/v3/endpoints` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:02.194990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ccc2625-7fa8-467d-8d9f-ca0e255f234b None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:02.196623 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ccc2625-7fa8-467d-8d9f-ca0e255f234b None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:02.221041 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 421/829] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:32:02 2022] POST /identity/v3/endpoints => generated 337 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:32:02.222275 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ccc2625-7fa8-467d-8d9f-ca0e255f234b None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:02.271455 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:02.365519 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ccc2625-7fa8-467d-8d9f-ca0e255f234b None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:02.400860 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a91627-ee1f-4855-a6c2-2a56b11bfbbe None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:02.415880 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d59342-3a1e-4bb8-b9f2-4f98e6719312 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:02.416438 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d59342-3a1e-4bb8-b9f2-4f98e6719312 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:02.416726 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d59342-3a1e-4bb8-b9f2-4f98e6719312 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:02.417008 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d59342-3a1e-4bb8-b9f2-4f98e6719312 None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:02.417713 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d59342-3a1e-4bb8-b9f2-4f98e6719312 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:02.418028 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d59342-3a1e-4bb8-b9f2-4f98e6719312 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:02.419695 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d59342-3a1e-4bb8-b9f2-4f98e6719312 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:02.434209 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 422/830] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Oct 28 09:32:02 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:02.440712 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d59342-3a1e-4bb8-b9f2-4f98e6719312 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:02.454706 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4db5f9a-7196-461e-a844-265d00bfdae0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:02.455434 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4db5f9a-7196-461e-a844-265d00bfdae0 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:02.455724 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4db5f9a-7196-461e-a844-265d00bfdae0 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:02.456063 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4db5f9a-7196-461e-a844-265d00bfdae0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:02.456954 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4db5f9a-7196-461e-a844-265d00bfdae0 None admin] RBAC: Adding query filter params (role.id=e13d798a98ea4c3dbb87a236ec683d32, scope.project.id=455d1c826b974bc38497d18a516867c7, user.id=688ceb201d304c2da5373c3cf36c2d79) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:02.457271 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4db5f9a-7196-461e-a844-265d00bfdae0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:02.458753 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4db5f9a-7196-461e-a844-265d00bfdae0 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:02.467449 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 409/831] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:32:02 2022] GET /identity/v3/role_assignments?user.id=688ceb201d304c2da5373c3cf36c2d79&scope.project.id=455d1c826b974bc38497d18a516867c7&role.id=e13d798a98ea4c3dbb87a236ec683d32 => generated 608 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:02.501861 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:02.583583 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d59342-3a1e-4bb8-b9f2-4f98e6719312 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:02.645087 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f454c5f-b213-45d1-b671-67c67f4f8034 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:02.645829 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f454c5f-b213-45d1-b671-67c67f4f8034 None admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:02.646109 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f454c5f-b213-45d1-b671-67c67f4f8034 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:02.646447 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f454c5f-b213-45d1-b671-67c67f4f8034 None admin] PATH_INFO: `/v3/projects/455d1c826b974bc38497d18a516867c7/users/97c8877c74194a6bbbf25b3c8f5b7dbd/roles/e5d8d090857a4a018d0ec3c79ece6472` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:02.666876 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f454c5f-b213-45d1-b671-67c67f4f8034 None admin] RBAC: Authorizing `identity:create_grant(project_id=455d1c826b974bc38497d18a516867c7, user_id=97c8877c74194a6bbbf25b3c8f5b7dbd, role_id=e5d8d090857a4a018d0ec3c79ece6472)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:02.668340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f454c5f-b213-45d1-b671-67c67f4f8034 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:02.686337 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 423/832] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Oct 28 09:32:02 2022] PUT /identity/v3/projects/455d1c826b974bc38497d18a516867c7/users/97c8877c74194a6bbbf25b3c8f5b7dbd/roles/e5d8d090857a4a018d0ec3c79ece6472 => generated 0 bytes in 246 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:32:03.113563 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299d268a-e2d5-4be4-ae5f-4e4225a39160 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:03.114129 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299d268a-e2d5-4be4-ae5f-4e4225a39160 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:03.114129 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299d268a-e2d5-4be4-ae5f-4e4225a39160 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:03.115718 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 410/833] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:32:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:32:03.121251 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f650214-c764-4273-aad7-9d9dbea7e441 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:03.121541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f650214-c764-4273-aad7-9d9dbea7e441 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:03.121823 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f650214-c764-4273-aad7-9d9dbea7e441 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:03.140783 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-2f650214-c764-4273-aad7-9d9dbea7e441 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:32:03.142032 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-2f650214-c764-4273-aad7-9d9dbea7e441 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:32:03.219679 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 424/834] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:32:03 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 09:32:03.486996 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-299d268a-e2d5-4be4-ae5f-4e4225a39160 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:03.603405 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-299d268a-e2d5-4be4-ae5f-4e4225a39160 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:03.619750 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:03.644056 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5aef570-b557-4e75-a540-ba50b29de219 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:03.644696 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5aef570-b557-4e75-a540-ba50b29de219 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:03.644980 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5aef570-b557-4e75-a540-ba50b29de219 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:03.645257 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5aef570-b557-4e75-a540-ba50b29de219 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:03.657452 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5aef570-b557-4e75-a540-ba50b29de219 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:32:03.662779 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-d5aef570-b557-4e75-a540-ba50b29de219 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:32:03.662779 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:03.662779 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:03.662779 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:03.662779 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:03.662779 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:03.662779 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:03.662779 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:03.662779 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:03.662779 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:03.662779 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:03.662779 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:03.662779 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:32:03.662779 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:32:03.662779 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:32:03.662779 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:32:03.663770 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:32:03.663770 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:32:03.663770 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:32:03.663770 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:32:03.663770 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:03.663770 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:03.663770 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:03.663770 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:03.663770 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:03.663770 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:03.663770 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:03.663770 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:03.663770 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:03.663770 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:03.663770 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:03.663770 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:03.664668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:03.664668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:03.664668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:03.664668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:03.664668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:32:03.664668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:32:03.664668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:32:03.664668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:32:03.664668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:32:03.664668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:32:03.664668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:32:03.664668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:03.664668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 411/835] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Oct 28 09:32:03 2022] GET /identity/v3/roles/admin => generated 83 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:03.671828 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f650214-c764-4273-aad7-9d9dbea7e441 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:03.691165 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:03.778727 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f650214-c764-4273-aad7-9d9dbea7e441 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:03.814172 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012ffab4-52ad-4d48-9231-065d64cce4e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:03.814819 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012ffab4-52ad-4d48-9231-065d64cce4e1 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:03.815101 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012ffab4-52ad-4d48-9231-065d64cce4e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:03.815375 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012ffab4-52ad-4d48-9231-065d64cce4e1 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:03.816039 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-012ffab4-52ad-4d48-9231-065d64cce4e1 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:03.816345 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-012ffab4-52ad-4d48-9231-065d64cce4e1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:03.817716 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-012ffab4-52ad-4d48-9231-065d64cce4e1 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:03.826943 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 425/836] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:32:03 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:03.833213 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5aef570-b557-4e75-a540-ba50b29de219 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:03.943882 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5aef570-b557-4e75-a540-ba50b29de219 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:03.979994 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65abaeb-74b6-4be2-8934-7873371bb5d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:03.980640 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65abaeb-74b6-4be2-8934-7873371bb5d3 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:03.980926 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65abaeb-74b6-4be2-8934-7873371bb5d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:03.981205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65abaeb-74b6-4be2-8934-7873371bb5d3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:03.991964 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a65abaeb-74b6-4be2-8934-7873371bb5d3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:03.993447 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a65abaeb-74b6-4be2-8934-7873371bb5d3 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:04.009282 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-a65abaeb-74b6-4be2-8934-7873371bb5d3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:32:04.009282 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:04.009282 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:04.009282 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:04.009282 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:04.009282 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:04.009282 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:04.009282 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:04.009282 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:04.009282 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:04.009282 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:04.009282 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:04.009282 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:32:04.009282 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:32:04.009282 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:32:04.009282 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:32:04.010326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:04.010326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:04.010326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:04.010326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:04.010326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:04.010326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:04.010326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:04.010326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:04.010326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:04.010326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:04.010326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:04.010326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:04.010326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:04.010326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:04.010326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:04.010326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:04.010326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:32:04.011279 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:32:04.011279 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:32:04.011279 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:04.011279 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 412/837] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:32:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:04.019695 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012ffab4-52ad-4d48-9231-065d64cce4e1 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:04.106079 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:04.127625 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012ffab4-52ad-4d48-9231-065d64cce4e1 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:04.167695 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc1b9ea-aa88-46cc-9b9d-d5eb061ae0a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:04.168343 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc1b9ea-aa88-46cc-9b9d-d5eb061ae0a1 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:04.168627 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc1b9ea-aa88-46cc-9b9d-d5eb061ae0a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:04.168903 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc1b9ea-aa88-46cc-9b9d-d5eb061ae0a1 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:04.169569 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc1b9ea-aa88-46cc-9b9d-d5eb061ae0a1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:04.169875 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc1b9ea-aa88-46cc-9b9d-d5eb061ae0a1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:04.171327 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbc1b9ea-aa88-46cc-9b9d-d5eb061ae0a1 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:04.185820 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 426/838] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:32:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:04.191723 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65abaeb-74b6-4be2-8934-7873371bb5d3 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:04.222091 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:04.296577 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5182fe16-0593-44c1-b209-cf5d6eb15e46 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:04.296928 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5182fe16-0593-44c1-b209-cf5d6eb15e46 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:04.297148 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5182fe16-0593-44c1-b209-cf5d6eb15e46 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:04.298643 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 427/839] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:32:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 09:32:04.304524 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b5df6c-e9df-46b9-8272-2261ddab0356 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:04.304814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b5df6c-e9df-46b9-8272-2261ddab0356 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:04.305101 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6b5df6c-e9df-46b9-8272-2261ddab0356 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:04.305391 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65abaeb-74b6-4be2-8934-7873371bb5d3 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:04.325804 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-d6b5df6c-e9df-46b9-8272-2261ddab0356 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:32:04.327037 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d6b5df6c-e9df-46b9-8272-2261ddab0356 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:32:04.347437 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e8e669c-067e-45e5-a961-6fb5b147d6fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:04.348094 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8e669c-067e-45e5-a961-6fb5b147d6fe None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:04.348405 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8e669c-067e-45e5-a961-6fb5b147d6fe None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:04.348646 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8e669c-067e-45e5-a961-6fb5b147d6fe None admin] PATH_INFO: `/v3/users/swift` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:04.357290 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e8e669c-067e-45e5-a961-6fb5b147d6fe None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:04.358724 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e8e669c-067e-45e5-a961-6fb5b147d6fe None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:04.371320 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-6e8e669c-067e-45e5-a961-6fb5b147d6fe None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Oct 28 09:32:04.371320 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:04.371320 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:04.371320 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:04.371320 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:04.371320 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:04.371320 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:04.371320 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:04.371320 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:04.371320 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:04.371320 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:04.371320 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:04.371320 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 09:32:04.371320 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 09:32:04.371320 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 09:32:04.371320 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 09:32:04.372331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:04.372331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:04.372331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 09:32:04.372331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:32:04.372331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 09:32:04.372331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:32:04.372331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:04.372331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:04.372331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:04.372331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:04.372331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:04.372331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:04.372331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:04.372331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:04.372331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:04.372331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:04.372331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:04.372331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:04.373368 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:04.373368 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:04.373368 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 09:32:04.373368 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 09:32:04.373368 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 09:32:04.373368 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 09:32:04.373368 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 09:32:04.373368 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 09:32:04.373368 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Oct 28 09:32:04.373368 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:04.377271 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 413/840] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Oct 28 09:32:04 2022] GET /identity/v3/users/swift => generated 83 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:04.378621 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e8e669c-067e-45e5-a961-6fb5b147d6fe None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:04.417757 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 428/841] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:32:04 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 09:32:04.474032 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:04.492802 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e8e669c-067e-45e5-a961-6fb5b147d6fe None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:04.528788 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2afeb804-277d-4c87-8f47-c3f825324e56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:04.529799 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2afeb804-277d-4c87-8f47-c3f825324e56 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:04.530284 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2afeb804-277d-4c87-8f47-c3f825324e56 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:04.530789 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2afeb804-277d-4c87-8f47-c3f825324e56 None admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:04.531838 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2afeb804-277d-4c87-8f47-c3f825324e56 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:04.532404 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2afeb804-277d-4c87-8f47-c3f825324e56 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:04.534293 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2afeb804-277d-4c87-8f47-c3f825324e56 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:04.553145 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 414/842] 10.222.0.25 () {56 vars in 1135 bytes} [Fri Oct 28 09:32:04 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:32:04.563642 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6b5df6c-e9df-46b9-8272-2261ddab0356 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:04.643786 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:04.677189 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6b5df6c-e9df-46b9-8272-2261ddab0356 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:04.713587 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13977734-84df-4259-b8da-f2bb19050564 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:04.714215 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13977734-84df-4259-b8da-f2bb19050564 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:04.714523 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13977734-84df-4259-b8da-f2bb19050564 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:04.714803 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13977734-84df-4259-b8da-f2bb19050564 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:04.723029 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2afeb804-277d-4c87-8f47-c3f825324e56 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:04.800830 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13977734-84df-4259-b8da-f2bb19050564 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:04.802934 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13977734-84df-4259-b8da-f2bb19050564 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:04.840091 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2afeb804-277d-4c87-8f47-c3f825324e56 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:04.882500 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a522b6dd-ae71-44b3-a903-310777d6f6bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:04.883142 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a522b6dd-ae71-44b3-a903-310777d6f6bf None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:04.883421 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a522b6dd-ae71-44b3-a903-310777d6f6bf None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:04.883692 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a522b6dd-ae71-44b3-a903-310777d6f6bf None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:04.884281 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a522b6dd-ae71-44b3-a903-310777d6f6bf None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:04.885633 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a522b6dd-ae71-44b3-a903-310777d6f6bf None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:04.887509 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:04.895740 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-a522b6dd-ae71-44b3-a903-310777d6f6bf None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 28 09:32:04.895740 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:04.895740 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 09:32:04.895740 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 09:32:04.895740 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 09:32:04.895740 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 09:32:04.895740 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 09:32:04.895740 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:32:04.895740 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 28 09:32:04.895740 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:04.895740 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 09:32:04.895740 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:04.895740 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:04.895740 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:04.895740 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:04.895740 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:04.895740 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:04.896810 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:04.896810 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:04.896810 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:04.896810 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:04.896810 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:04.896810 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:04.896810 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 09:32:04.896810 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 09:32:04.896810 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 09:32:04.896810 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 09:32:04.896810 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:04.896810 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:04.896810 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:04.896810 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:04.896810 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:04.896810 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:04.897725 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:04.897725 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:04.897725 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:04.897725 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:04.897725 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:04.897725 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:04.897725 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:04.897725 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:04.897725 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:04.897725 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:04.897725 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 09:32:04.897725 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:32:04.897725 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 28 09:32:04.897725 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:04.901633 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 415/843] 10.222.0.25 () {56 vars in 1114 bytes} [Fri Oct 28 09:32:04 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:32:04.902893 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6cedf4-d67c-494a-9ca5-3f79be2c1a06 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:04.903179 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6cedf4-d67c-494a-9ca5-3f79be2c1a06 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:04.903450 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6cedf4-d67c-494a-9ca5-3f79be2c1a06 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:04.909056 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 416/844] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:32:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:32:04.909056 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a6cedf4-d67c-494a-9ca5-3f79be2c1a06 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:04.919382 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:04.931382 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 429/845] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:32:04 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 09:32:04.932650 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f3dc3f-7113-4b25-9556-4c84aa9b79e9 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:04.932938 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f3dc3f-7113-4b25-9556-4c84aa9b79e9 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:04.933228 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f3dc3f-7113-4b25-9556-4c84aa9b79e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:04.954012 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-28f3dc3f-7113-4b25-9556-4c84aa9b79e9 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:32:04.955241 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-28f3dc3f-7113-4b25-9556-4c84aa9b79e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:32:05.016785 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a6cedf4-d67c-494a-9ca5-3f79be2c1a06 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:05.029900 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 430/846] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:32:04 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:32:05.030999 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f3dc3f-7113-4b25-9556-4c84aa9b79e9 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:05.054446 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bca2e9f-c7e1-45ee-b321-17f1687ba328 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:05.055066 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bca2e9f-c7e1-45ee-b321-17f1687ba328 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:05.055337 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bca2e9f-c7e1-45ee-b321-17f1687ba328 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:05.055609 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bca2e9f-c7e1-45ee-b321-17f1687ba328 None admin] PATH_INFO: `/v3/services` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:05.056236 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bca2e9f-c7e1-45ee-b321-17f1687ba328 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:05.056934 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bca2e9f-c7e1-45ee-b321-17f1687ba328 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:05.058032 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bca2e9f-c7e1-45ee-b321-17f1687ba328 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:05.067665 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 417/847] 10.222.0.25 () {56 vars in 1114 bytes} [Fri Oct 28 09:32:04 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:05.067665 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469e4360-0f26-4be4-b78a-2fa593e9495e None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:05.067665 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469e4360-0f26-4be4-b78a-2fa593e9495e None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:05.067665 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469e4360-0f26-4be4-b78a-2fa593e9495e None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:05.067665 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 418/848] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:32:05 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:32:05.068566 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-469e4360-0f26-4be4-b78a-2fa593e9495e None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:05.144578 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:05.148484 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f3dc3f-7113-4b25-9556-4c84aa9b79e9 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:05.148633 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:05.181929 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-469e4360-0f26-4be4-b78a-2fa593e9495e None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:05.184723 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc72a919-4927-4ce6-a390-50f884e73bc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:05.185386 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc72a919-4927-4ce6-a390-50f884e73bc4 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:05.185671 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc72a919-4927-4ce6-a390-50f884e73bc4 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:05.185950 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc72a919-4927-4ce6-a390-50f884e73bc4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:05.196295 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc72a919-4927-4ce6-a390-50f884e73bc4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:05.197722 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc72a919-4927-4ce6-a390-50f884e73bc4 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:05.213275 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-fc72a919-4927-4ce6-a390-50f884e73bc4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:32:05.213275 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:05.213275 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:05.213275 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:05.213275 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:05.213275 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:05.213275 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:05.213275 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:05.213275 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:05.213275 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:05.213275 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:05.213275 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:05.213275 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:32:05.213275 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:32:05.213275 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:32:05.213275 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:32:05.214223 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:05.214223 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:05.214223 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:05.214223 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:05.214223 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:05.214223 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:05.214223 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:05.214223 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:05.214223 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:05.214223 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:05.214223 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:05.214223 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:05.214223 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:05.214223 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:05.214223 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:05.214223 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:05.214223 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:32:05.215153 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:32:05.215153 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:32:05.215153 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:05.215153 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 431/849] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:32:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:05.215653 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45ac6df-8402-46a4-b219-6e264d2d4643 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:05.215919 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45ac6df-8402-46a4-b219-6e264d2d4643 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:05.216194 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f45ac6df-8402-46a4-b219-6e264d2d4643 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:05.223404 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7aaf05e-9cad-456e-bf22-22d003ddbb87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:05.224020 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7aaf05e-9cad-456e-bf22-22d003ddbb87 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:05.224300 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7aaf05e-9cad-456e-bf22-22d003ddbb87 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:05.224573 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7aaf05e-9cad-456e-bf22-22d003ddbb87 None admin] PATH_INFO: `/v3/services` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:05.225254 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7aaf05e-9cad-456e-bf22-22d003ddbb87 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:05.226082 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7aaf05e-9cad-456e-bf22-22d003ddbb87 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:05.227487 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7aaf05e-9cad-456e-bf22-22d003ddbb87 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:05.242627 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 419/850] 10.222.0.25 () {56 vars in 1114 bytes} [Fri Oct 28 09:32:05 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:05.242627 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7aaf05e-9cad-456e-bf22-22d003ddbb87 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:05.243786 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-f45ac6df-8402-46a4-b219-6e264d2d4643 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:32:05.245049 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-f45ac6df-8402-46a4-b219-6e264d2d4643 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:32:05.330766 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 432/851] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:32:05 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:32:05.331793 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f45ac6df-8402-46a4-b219-6e264d2d4643 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:05.375043 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7aaf05e-9cad-456e-bf22-22d003ddbb87 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:05.426605 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b5e9bc-166a-406f-b4a2-3f3c5aa46ccb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:05.427495 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b5e9bc-166a-406f-b4a2-3f3c5aa46ccb None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:05.427862 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b5e9bc-166a-406f-b4a2-3f3c5aa46ccb None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:05.428140 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b5e9bc-166a-406f-b4a2-3f3c5aa46ccb None admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:05.434116 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1b5e9bc-166a-406f-b4a2-3f3c5aa46ccb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:05.434116 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1b5e9bc-166a-406f-b4a2-3f3c5aa46ccb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:05.434116 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1b5e9bc-166a-406f-b4a2-3f3c5aa46ccb None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:05.434642 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:05.443514 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:05.449075 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 420/852] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:32:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:05.454892 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b5e9bc-166a-406f-b4a2-3f3c5aa46ccb None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:05.469689 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f45ac6df-8402-46a4-b219-6e264d2d4643 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:05.508583 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03283cb4-49aa-4317-9029-1277eadef1a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:05.509282 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03283cb4-49aa-4317-9029-1277eadef1a5 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:05.509567 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03283cb4-49aa-4317-9029-1277eadef1a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:05.509837 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03283cb4-49aa-4317-9029-1277eadef1a5 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:05.510524 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03283cb4-49aa-4317-9029-1277eadef1a5 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:05.511944 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03283cb4-49aa-4317-9029-1277eadef1a5 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:05.522305 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-03283cb4-49aa-4317-9029-1277eadef1a5 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 28 09:32:05.522305 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:05.522305 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 09:32:05.522305 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 09:32:05.522305 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 09:32:05.522305 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 09:32:05.522305 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 09:32:05.522305 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:32:05.522305 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 28 09:32:05.522305 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:05.522305 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 09:32:05.522305 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:05.522305 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:05.522305 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:05.522305 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:05.522305 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:05.523372 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:05.523372 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:05.523372 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:05.523372 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:05.523372 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:05.523372 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:05.523372 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:05.523372 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 09:32:05.523372 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 09:32:05.523372 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 09:32:05.523372 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 09:32:05.523372 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:05.523372 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:05.523372 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:05.523372 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:05.523372 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:05.523372 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:05.524276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:05.524276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:05.524276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:05.524276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:05.524276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:05.524276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:05.524276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:05.524276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:05.524276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:05.524276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:05.524276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 09:32:05.524276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:32:05.524276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 28 09:32:05.524276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:05.524276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 433/853] 10.222.0.25 () {56 vars in 1132 bytes} [Fri Oct 28 09:32:05 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:05.529941 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03283cb4-49aa-4317-9029-1277eadef1a5 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:05.597960 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b5e9bc-166a-406f-b4a2-3f3c5aa46ccb None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:05.637815 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17836c76-a932-4607-a8da-bcd4e7948775 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:05.638732 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17836c76-a932-4607-a8da-bcd4e7948775 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:05.639021 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17836c76-a932-4607-a8da-bcd4e7948775 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:05.639719 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17836c76-a932-4607-a8da-bcd4e7948775 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:05.649137 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03283cb4-49aa-4317-9029-1277eadef1a5 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:05.649382 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17836c76-a932-4607-a8da-bcd4e7948775 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:05.650875 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17836c76-a932-4607-a8da-bcd4e7948775 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:05.664982 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-17836c76-a932-4607-a8da-bcd4e7948775 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:32:05.664982 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:05.664982 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:05.664982 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:05.664982 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:05.664982 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:05.664982 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:05.664982 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:05.664982 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:05.664982 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:05.664982 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:05.664982 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:05.664982 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 09:32:05.664982 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 09:32:05.664982 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 09:32:05.666110 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 09:32:05.666110 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:05.666110 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:05.666110 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:05.666110 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:05.666110 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:05.666110 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:05.666110 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:05.666110 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:05.666110 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:05.666110 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:05.666110 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:05.666110 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:05.666110 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:05.666110 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:05.666110 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:05.666110 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:05.667183 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 09:32:05.667183 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 09:32:05.667183 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 09:32:05.667183 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 09:32:05.667183 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 09:32:05.667183 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 09:32:05.667183 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:32:05.667183 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:05.667183 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 421/854] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:32:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:32:05.667183 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17836c76-a932-4607-a8da-bcd4e7948775 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:05.693417 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba3c6f2e-5203-4f29-b852-d7aceb4b3cea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:05.694655 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3c6f2e-5203-4f29-b852-d7aceb4b3cea None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:05.695026 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3c6f2e-5203-4f29-b852-d7aceb4b3cea None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:05.695201 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba3c6f2e-5203-4f29-b852-d7aceb4b3cea None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:05.696028 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba3c6f2e-5203-4f29-b852-d7aceb4b3cea None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:05.696204 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba3c6f2e-5203-4f29-b852-d7aceb4b3cea None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:05.697599 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba3c6f2e-5203-4f29-b852-d7aceb4b3cea None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:05.709102 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 434/855] 10.222.0.25 () {56 vars in 1126 bytes} [Fri Oct 28 09:32:05 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:05.709102 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba3c6f2e-5203-4f29-b852-d7aceb4b3cea None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:05.726448 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:05.731242 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:05.778778 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17836c76-a932-4607-a8da-bcd4e7948775 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:05.816226 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4372ceb-63e9-4977-acf5-07af60cd2ece None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:05.816861 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4372ceb-63e9-4977-acf5-07af60cd2ece None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:05.817141 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4372ceb-63e9-4977-acf5-07af60cd2ece None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:05.817414 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4372ceb-63e9-4977-acf5-07af60cd2ece None admin] PATH_INFO: `/v3/roles/service` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:05.829222 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4372ceb-63e9-4977-acf5-07af60cd2ece None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:32:05.829709 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba3c6f2e-5203-4f29-b852-d7aceb4b3cea None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:05.834200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-d4372ceb-63e9-4977-acf5-07af60cd2ece None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 09:32:05.834200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:05.834200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:05.834200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:05.834200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:05.834200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:05.834200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:05.834200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:05.834200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:05.834200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:05.834200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:05.834200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:05.834200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:32:05.834200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:32:05.834200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:32:05.834200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:32:05.835215 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:32:05.835215 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:32:05.835215 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:32:05.835215 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:32:05.835215 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:05.835215 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:05.835215 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:05.835215 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:05.835215 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:05.835215 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:05.835215 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:05.835215 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:05.835215 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:05.835215 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:05.835215 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:05.835215 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:05.836061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:05.836061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:05.836061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:05.836061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:05.836061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:32:05.836061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:32:05.836061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:32:05.836061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:32:05.836061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:32:05.836061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:32:05.836061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 09:32:05.836061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:05.836061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 422/856] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Oct 28 09:32:05 2022] GET /identity/v3/roles/service => generated 85 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:05.836821 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4372ceb-63e9-4977-acf5-07af60cd2ece None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:05.867891 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9433d8de-3e48-4f60-b968-d39df7fd4d26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:05.868609 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9433d8de-3e48-4f60-b968-d39df7fd4d26 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:05.868957 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9433d8de-3e48-4f60-b968-d39df7fd4d26 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:05.869239 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9433d8de-3e48-4f60-b968-d39df7fd4d26 None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:05.869935 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9433d8de-3e48-4f60-b968-d39df7fd4d26 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:05.870242 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9433d8de-3e48-4f60-b968-d39df7fd4d26 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:05.871807 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9433d8de-3e48-4f60-b968-d39df7fd4d26 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:05.884119 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 435/857] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Oct 28 09:32:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:05.884955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9433d8de-3e48-4f60-b968-d39df7fd4d26 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:05.953659 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4372ceb-63e9-4977-acf5-07af60cd2ece None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:05.959954 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:05.970863 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:05.992234 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85df5e7b-cdb0-4e0e-a693-f7141bc95317 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:05.992859 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85df5e7b-cdb0-4e0e-a693-f7141bc95317 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:05.993144 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85df5e7b-cdb0-4e0e-a693-f7141bc95317 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:05.993465 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85df5e7b-cdb0-4e0e-a693-f7141bc95317 None admin] PATH_INFO: `/v3/services` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:05.994063 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85df5e7b-cdb0-4e0e-a693-f7141bc95317 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:05.994379 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85df5e7b-cdb0-4e0e-a693-f7141bc95317 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:05.995798 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85df5e7b-cdb0-4e0e-a693-f7141bc95317 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:05.996452 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9433d8de-3e48-4f60-b968-d39df7fd4d26 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:06.007092 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 423/858] 10.222.0.25 () {56 vars in 1126 bytes} [Fri Oct 28 09:32:05 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:06.007092 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85df5e7b-cdb0-4e0e-a693-f7141bc95317 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:06.039168 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ff9b67-ae61-40d5-b80a-66c4c6b10745 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:06.039893 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ff9b67-ae61-40d5-b80a-66c4c6b10745 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:06.040248 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ff9b67-ae61-40d5-b80a-66c4c6b10745 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:06.040595 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ff9b67-ae61-40d5-b80a-66c4c6b10745 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:06.041406 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2ff9b67-ae61-40d5-b80a-66c4c6b10745 None admin] RBAC: Adding query filter params (name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:06.041788 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2ff9b67-ae61-40d5-b80a-66c4c6b10745 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:06.043281 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2ff9b67-ae61-40d5-b80a-66c4c6b10745 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:06.052389 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 436/859] 10.222.0.25 () {56 vars in 1103 bytes} [Fri Oct 28 09:32:05 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:06.058766 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ff9b67-ae61-40d5-b80a-66c4c6b10745 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:06.161103 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85df5e7b-cdb0-4e0e-a693-f7141bc95317 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:06.179477 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ff9b67-ae61-40d5-b80a-66c4c6b10745 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:06.212363 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5118079-0f4f-4280-b76e-3f3a91ded82c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:06.212926 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5118079-0f4f-4280-b76e-3f3a91ded82c None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:06.213213 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5118079-0f4f-4280-b76e-3f3a91ded82c None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:06.213643 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5118079-0f4f-4280-b76e-3f3a91ded82c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:06.214413 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5118079-0f4f-4280-b76e-3f3a91ded82c None admin] RBAC: Adding query filter params (role.id=e13d798a98ea4c3dbb87a236ec683d32, scope.project.id=455d1c826b974bc38497d18a516867c7, user.id=9c6cffe0d56e45c4b8fb54fbe5df6094) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:06.215853 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5118079-0f4f-4280-b76e-3f3a91ded82c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:06.216563 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5118079-0f4f-4280-b76e-3f3a91ded82c None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:06.218918 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:06.223437 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65859ba-4d85-48fa-8275-51a7a44a0b9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:06.224011 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65859ba-4d85-48fa-8275-51a7a44a0b9c None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:06.224305 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65859ba-4d85-48fa-8275-51a7a44a0b9c None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:06.224586 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65859ba-4d85-48fa-8275-51a7a44a0b9c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:06.224686 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 424/860] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:32:06 2022] GET /identity/v3/role_assignments?user.id=9c6cffe0d56e45c4b8fb54fbe5df6094&scope.project.id=455d1c826b974bc38497d18a516867c7&role.id=e13d798a98ea4c3dbb87a236ec683d32 => generated 608 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:06.239311 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d65859ba-4d85-48fa-8275-51a7a44a0b9c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:06.241173 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d65859ba-4d85-48fa-8275-51a7a44a0b9c None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:06.258486 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-d65859ba-4d85-48fa-8275-51a7a44a0b9c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:32:06.258486 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:06.258486 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:06.258486 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:06.258486 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:06.258486 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:06.258486 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:06.258486 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:06.258486 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:06.258486 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:06.258486 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:06.258486 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:06.258486 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:32:06.258486 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:32:06.258486 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:32:06.258486 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:32:06.259699 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:06.259699 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:06.259699 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:06.259699 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:06.259699 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:06.259699 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:06.259699 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:06.259699 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:06.259699 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:06.259699 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:06.259699 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:06.259699 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:06.259699 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:06.259699 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:06.259699 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:06.259699 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:06.259699 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:32:06.260802 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:32:06.260802 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:32:06.260802 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:06.260802 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 437/861] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:32:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:06.265619 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5118079-0f4f-4280-b76e-3f3a91ded82c None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:06.322143 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:06.393698 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5118079-0f4f-4280-b76e-3f3a91ded82c None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:06.436948 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-904a122b-0ff4-432b-9921-22bc383137e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:06.437557 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904a122b-0ff4-432b-9921-22bc383137e7 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:06.437842 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904a122b-0ff4-432b-9921-22bc383137e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:06.438116 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904a122b-0ff4-432b-9921-22bc383137e7 None admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:06.438800 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-904a122b-0ff4-432b-9921-22bc383137e7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:06.439102 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-904a122b-0ff4-432b-9921-22bc383137e7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:06.440479 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-904a122b-0ff4-432b-9921-22bc383137e7 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:06.453628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 425/862] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:32:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:06.458874 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65859ba-4d85-48fa-8275-51a7a44a0b9c None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:06.581900 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65859ba-4d85-48fa-8275-51a7a44a0b9c None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:06.617750 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c002fc-9952-4899-98b8-9a23923e148f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:06.618410 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c002fc-9952-4899-98b8-9a23923e148f None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:06.618708 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c002fc-9952-4899-98b8-9a23923e148f None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:06.619125 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c002fc-9952-4899-98b8-9a23923e148f None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:06.626955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54c002fc-9952-4899-98b8-9a23923e148f None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:06.628394 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54c002fc-9952-4899-98b8-9a23923e148f None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:06.640202 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-54c002fc-9952-4899-98b8-9a23923e148f None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 28 09:32:06.640202 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:06.640202 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:06.640202 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:06.640202 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:06.640202 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:06.640202 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:06.640202 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:06.640202 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:06.640202 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:06.640202 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:06.640202 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:06.640202 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 09:32:06.640202 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 09:32:06.640202 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 09:32:06.640202 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 09:32:06.641168 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:06.641168 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:06.641168 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 09:32:06.641168 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:32:06.641168 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 09:32:06.641168 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:32:06.641168 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:06.641168 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:06.641168 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:06.641168 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:06.641168 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:06.641168 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:06.641168 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:06.641168 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:06.641168 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:06.641168 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:06.641168 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:06.641168 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:06.642110 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:06.642110 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:06.642110 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 09:32:06.642110 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 09:32:06.642110 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 09:32:06.642110 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 09:32:06.642110 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 09:32:06.642110 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 09:32:06.642110 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 28 09:32:06.642110 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:06.642110 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 438/863] 10.222.0.25 () {56 vars in 1117 bytes} [Fri Oct 28 09:32:06 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:06.647584 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-904a122b-0ff4-432b-9921-22bc383137e7 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:06.751687 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-904a122b-0ff4-432b-9921-22bc383137e7 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:06.754942 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:06.789880 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b58984f-94d9-457e-9664-e8838cbaf4e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:06.790538 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b58984f-94d9-457e-9664-e8838cbaf4e1 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:06.790826 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b58984f-94d9-457e-9664-e8838cbaf4e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:06.791105 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b58984f-94d9-457e-9664-e8838cbaf4e1 None admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:06.791841 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b58984f-94d9-457e-9664-e8838cbaf4e1 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:06.792152 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b58984f-94d9-457e-9664-e8838cbaf4e1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:06.793579 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b58984f-94d9-457e-9664-e8838cbaf4e1 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:06.812781 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 426/864] 10.222.0.25 () {56 vars in 1149 bytes} [Fri Oct 28 09:32:06 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:06.819638 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c002fc-9952-4899-98b8-9a23923e148f None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:06.869767 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:06.927921 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c002fc-9952-4899-98b8-9a23923e148f None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:06.963470 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3e83e6-b4c9-48ab-a61d-dfdd36b86821 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:06.964330 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3e83e6-b4c9-48ab-a61d-dfdd36b86821 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:06.964762 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3e83e6-b4c9-48ab-a61d-dfdd36b86821 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:06.965184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3e83e6-b4c9-48ab-a61d-dfdd36b86821 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:07.042525 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae3e83e6-b4c9-48ab-a61d-dfdd36b86821 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:07.044347 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae3e83e6-b4c9-48ab-a61d-dfdd36b86821 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:07.094061 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:07.153390 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 439/865] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:32:06 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 09:32:07.159531 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b58984f-94d9-457e-9664-e8838cbaf4e1 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:07.273830 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b58984f-94d9-457e-9664-e8838cbaf4e1 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:07.312835 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42cff68-74f7-4d5e-80ef-10b572b9f4e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:07.313485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42cff68-74f7-4d5e-80ef-10b572b9f4e2 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:07.313767 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42cff68-74f7-4d5e-80ef-10b572b9f4e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:07.314043 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42cff68-74f7-4d5e-80ef-10b572b9f4e2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:07.324668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42cff68-74f7-4d5e-80ef-10b572b9f4e2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:07.326180 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42cff68-74f7-4d5e-80ef-10b572b9f4e2 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:07.341932 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-f42cff68-74f7-4d5e-80ef-10b572b9f4e2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:32:07.341932 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:07.341932 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:07.341932 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:07.341932 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:07.341932 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:07.341932 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:07.341932 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:07.341932 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:07.341932 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:07.341932 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:07.341932 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:07.341932 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:32:07.341932 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:32:07.341932 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:32:07.341932 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:32:07.342930 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:07.342930 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:07.342930 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:07.342930 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:07.342930 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:07.342930 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:07.342930 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:07.342930 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:07.342930 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:07.342930 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:07.342930 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:07.342930 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:07.342930 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:07.342930 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:07.342930 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:07.342930 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:07.342930 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:32:07.343816 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:32:07.343816 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:32:07.343816 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:07.343816 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 427/866] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:32:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:07.351568 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3e83e6-b4c9-48ab-a61d-dfdd36b86821 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:07.458774 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3e83e6-b4c9-48ab-a61d-dfdd36b86821 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:07.494115 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e2a3bd-2775-49de-90d1-be40477e30ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:07.494820 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e2a3bd-2775-49de-90d1-be40477e30ff None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:07.495105 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e2a3bd-2775-49de-90d1-be40477e30ff None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:07.495378 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e2a3bd-2775-49de-90d1-be40477e30ff None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:07.496021 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e2a3bd-2775-49de-90d1-be40477e30ff None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:07.496325 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e2a3bd-2775-49de-90d1-be40477e30ff None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:07.497688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e2a3bd-2775-49de-90d1-be40477e30ff None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:07.511405 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 440/867] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:32:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:07.517141 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42cff68-74f7-4d5e-80ef-10b572b9f4e2 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:07.537022 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:07.555671 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3fe9b4-db50-45bc-b7d3-8bfda5c1b2ff None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:07.556016 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3fe9b4-db50-45bc-b7d3-8bfda5c1b2ff None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:07.556297 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3fe9b4-db50-45bc-b7d3-8bfda5c1b2ff None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:07.557857 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 441/868] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:32:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 09:32:07.564701 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4eb20a-8cd4-4ad0-9279-7a0bf0d573ae None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:07.565018 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4eb20a-8cd4-4ad0-9279-7a0bf0d573ae None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:07.565378 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4eb20a-8cd4-4ad0-9279-7a0bf0d573ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:07.574013 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:07.591405 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-6f4eb20a-8cd4-4ad0-9279-7a0bf0d573ae None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:32:07.593189 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-6f4eb20a-8cd4-4ad0-9279-7a0bf0d573ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:32:07.640601 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42cff68-74f7-4d5e-80ef-10b572b9f4e2 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:07.680964 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b218f7-1742-49eb-ab1d-5f2210e6bd68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:07.681762 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b218f7-1742-49eb-ab1d-5f2210e6bd68 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:07.681942 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 442/869] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:32:07 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 09:32:07.682729 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b218f7-1742-49eb-ab1d-5f2210e6bd68 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:07.683023 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b218f7-1742-49eb-ab1d-5f2210e6bd68 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:07.702607 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86b218f7-1742-49eb-ab1d-5f2210e6bd68 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:07.704096 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86b218f7-1742-49eb-ab1d-5f2210e6bd68 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:07.716649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-86b218f7-1742-49eb-ab1d-5f2210e6bd68 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:32:07.716649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:07.716649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:07.716649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:07.716649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:07.716649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:07.716649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:07.716649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:07.716649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:07.716649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:07.716649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:07.716649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:07.716649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 09:32:07.716649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 09:32:07.716649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 09:32:07.717740 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 09:32:07.717740 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:07.717740 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:07.717740 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:07.717740 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:07.717740 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:07.717740 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:07.717740 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:07.717740 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:07.717740 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:07.717740 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:07.717740 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:07.717740 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:07.717740 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:07.717740 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:07.717740 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:07.717740 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:07.718788 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 09:32:07.718788 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 09:32:07.718788 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 09:32:07.718788 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 09:32:07.718788 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 09:32:07.718788 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 09:32:07.718788 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:32:07.718788 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:07.718788 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 428/870] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:32:07 2022] GET /identity/v3/projects/service => generated 88 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:07.725825 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f4eb20a-8cd4-4ad0-9279-7a0bf0d573ae None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:07.839003 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f4eb20a-8cd4-4ad0-9279-7a0bf0d573ae None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:07.883742 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3de799-ae28-422c-9491-eeee24d37536 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:07.884357 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3de799-ae28-422c-9491-eeee24d37536 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:07.884634 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3de799-ae28-422c-9491-eeee24d37536 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:07.884907 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3de799-ae28-422c-9491-eeee24d37536 None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:07.885600 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e3de799-ae28-422c-9491-eeee24d37536 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:07.885902 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e3de799-ae28-422c-9491-eeee24d37536 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:07.887352 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e3de799-ae28-422c-9491-eeee24d37536 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:07.889059 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:07.901250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 443/871] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Oct 28 09:32:07 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:07.906989 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b218f7-1742-49eb-ab1d-5f2210e6bd68 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:07.963672 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3de799-ae28-422c-9491-eeee24d37536 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:07.991295 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:08.017551 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b218f7-1742-49eb-ab1d-5f2210e6bd68 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:08.056011 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04341535-5860-4692-bb8e-a8d9fe6e413d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:08.056658 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04341535-5860-4692-bb8e-a8d9fe6e413d None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:08.057271 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04341535-5860-4692-bb8e-a8d9fe6e413d None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:08.057551 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04341535-5860-4692-bb8e-a8d9fe6e413d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:08.058640 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04341535-5860-4692-bb8e-a8d9fe6e413d None admin] RBAC: Adding query filter params (role.id=e5d8d090857a4a018d0ec3c79ece6472, scope.project.id=455d1c826b974bc38497d18a516867c7, user.id=97c8877c74194a6bbbf25b3c8f5b7dbd) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:08.059003 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04341535-5860-4692-bb8e-a8d9fe6e413d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:08.060942 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04341535-5860-4692-bb8e-a8d9fe6e413d None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:08.074709 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 429/872] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:32:07 2022] GET /identity/v3/role_assignments?user.id=97c8877c74194a6bbbf25b3c8f5b7dbd&scope.project.id=455d1c826b974bc38497d18a516867c7&role.id=e5d8d090857a4a018d0ec3c79ece6472 => generated 608 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:08.082992 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3de799-ae28-422c-9491-eeee24d37536 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:08.127753 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb2c6a9-4adc-4a81-b7f8-82980786e17d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:08.128929 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb2c6a9-4adc-4a81-b7f8-82980786e17d None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:08.129420 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb2c6a9-4adc-4a81-b7f8-82980786e17d None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:08.129914 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb2c6a9-4adc-4a81-b7f8-82980786e17d None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:08.130869 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eb2c6a9-4adc-4a81-b7f8-82980786e17d None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:08.132621 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eb2c6a9-4adc-4a81-b7f8-82980786e17d None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:08.151987 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 444/873] 10.222.0.25 () {60 vars in 1141 bytes} [Fri Oct 28 09:32:07 2022] POST /identity/v3/services => generated 251 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:32:08.327160 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37abfd6b-dc50-4097-82f8-67711cdc8861 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:08.327658 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37abfd6b-dc50-4097-82f8-67711cdc8861 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:08.327658 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37abfd6b-dc50-4097-82f8-67711cdc8861 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:08.329345 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 430/874] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:32:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:32:08.335717 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebcf1f0-00d6-42bf-8ead-c08bd048cef7 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:08.336011 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebcf1f0-00d6-42bf-8ead-c08bd048cef7 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:08.336298 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebcf1f0-00d6-42bf-8ead-c08bd048cef7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:08.357745 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-5ebcf1f0-00d6-42bf-8ead-c08bd048cef7 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:32:08.358996 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-5ebcf1f0-00d6-42bf-8ead-c08bd048cef7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:32:08.419566 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:08.448009 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 445/875] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:32:08 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:32:08.726839 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37abfd6b-dc50-4097-82f8-67711cdc8861 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:08.736313 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8df83b-e575-491f-94e0-8a31ca5c687b None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:08.736634 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8df83b-e575-491f-94e0-8a31ca5c687b None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:08.736867 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8df83b-e575-491f-94e0-8a31ca5c687b None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:08.738194 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 446/876] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:32:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 09:32:08.745286 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39bb505-46d1-4db4-a8b4-f491f51171e8 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:08.745566 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39bb505-46d1-4db4-a8b4-f491f51171e8 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:08.745837 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39bb505-46d1-4db4-a8b4-f491f51171e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:08.765290 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-a39bb505-46d1-4db4-a8b4-f491f51171e8 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:32:08.766527 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a39bb505-46d1-4db4-a8b4-f491f51171e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:32:08.848323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 447/877] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:32:08 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 09:32:08.867010 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37abfd6b-dc50-4097-82f8-67711cdc8861 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:08.904802 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64e1fc03-8c5f-4694-ab4c-5f42c1fcbac0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:08.905428 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e1fc03-8c5f-4694-ab4c-5f42c1fcbac0 None admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:08.905709 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e1fc03-8c5f-4694-ab4c-5f42c1fcbac0 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:08.905983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e1fc03-8c5f-4694-ab4c-5f42c1fcbac0 None admin] PATH_INFO: `/v3/services` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:08.906693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64e1fc03-8c5f-4694-ab4c-5f42c1fcbac0 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:08.908329 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64e1fc03-8c5f-4694-ab4c-5f42c1fcbac0 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:08.920881 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:08.931666 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 431/878] 10.222.0.25 () {60 vars in 1141 bytes} [Fri Oct 28 09:32:08 2022] POST /identity/v3/services => generated 269 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:32:09.126809 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a39bb505-46d1-4db4-a8b4-f491f51171e8 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:09.246240 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a39bb505-46d1-4db4-a8b4-f491f51171e8 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:09.258158 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:09.289738 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc7c68e-85f4-4a04-bd73-d825bca2bd86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:09.290379 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc7c68e-85f4-4a04-bd73-d825bca2bd86 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:09.290689 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc7c68e-85f4-4a04-bd73-d825bca2bd86 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:09.290965 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc7c68e-85f4-4a04-bd73-d825bca2bd86 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:09.291549 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc7c68e-85f4-4a04-bd73-d825bca2bd86 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:09.292906 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc7c68e-85f4-4a04-bd73-d825bca2bd86 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:09.306242 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-5fc7c68e-85f4-4a04-bd73-d825bca2bd86 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 09:32:09.306242 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:09.306242 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 09:32:09.306242 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 09:32:09.306242 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 09:32:09.306242 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 09:32:09.306242 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 09:32:09.306242 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:32:09.306242 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 09:32:09.306242 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:09.306242 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 09:32:09.306242 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:09.306242 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:09.306242 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:09.306242 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:09.306242 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:09.306242 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:09.307298 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:09.307298 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:09.307298 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:09.307298 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:09.307298 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:09.307298 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:09.307298 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 09:32:09.307298 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 09:32:09.307298 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 09:32:09.307298 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 09:32:09.307298 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:09.307298 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:09.307298 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:09.307298 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:09.307298 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:09.307298 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:09.308134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:09.308134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:09.308134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:09.308134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:09.308134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:09.308134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:09.308134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:09.308134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:09.308134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:09.308134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:09.308134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 09:32:09.308134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:32:09.308134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 09:32:09.308134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:09.308134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 448/879] 10.222.0.25 () {56 vars in 1126 bytes} [Fri Oct 28 09:32:09 2022] GET /identity/v3/services/object-store => generated 93 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:09.313556 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64e1fc03-8c5f-4694-ab4c-5f42c1fcbac0 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:09.427100 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64e1fc03-8c5f-4694-ab4c-5f42c1fcbac0 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:09.465687 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ed17b7-772f-4ade-acc7-a73c41fd01bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:09.466306 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ed17b7-772f-4ade-acc7-a73c41fd01bc None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:09.466617 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ed17b7-772f-4ade-acc7-a73c41fd01bc None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:09.466933 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ed17b7-772f-4ade-acc7-a73c41fd01bc None admin] PATH_INFO: `/v3/services` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:09.467644 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ed17b7-772f-4ade-acc7-a73c41fd01bc None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:09.467955 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ed17b7-772f-4ade-acc7-a73c41fd01bc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:09.469499 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ed17b7-772f-4ade-acc7-a73c41fd01bc None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:09.474917 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 432/880] 10.222.0.25 () {56 vars in 1122 bytes} [Fri Oct 28 09:32:09 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:09.480608 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc7c68e-85f4-4a04-bd73-d825bca2bd86 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:09.620544 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc7c68e-85f4-4a04-bd73-d825bca2bd86 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:09.655713 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6338ab0d-e057-4fe4-9b5a-e539943db6ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:09.656378 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6338ab0d-e057-4fe4-9b5a-e539943db6ed None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:09.656659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6338ab0d-e057-4fe4-9b5a-e539943db6ed None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:09.656984 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6338ab0d-e057-4fe4-9b5a-e539943db6ed None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:09.657620 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6338ab0d-e057-4fe4-9b5a-e539943db6ed None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:09.657923 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6338ab0d-e057-4fe4-9b5a-e539943db6ed None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:09.659564 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6338ab0d-e057-4fe4-9b5a-e539943db6ed None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:09.665351 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 449/881] 10.222.0.25 () {56 vars in 1122 bytes} [Fri Oct 28 09:32:09 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:10.472534 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307ca8b9-521a-4f6b-b48b-87208541a71c None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:10.473584 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307ca8b9-521a-4f6b-b48b-87208541a71c None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:10.474078 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307ca8b9-521a-4f6b-b48b-87208541a71c None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:10.476101 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 433/882] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:32:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:32:10.481825 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e0fe81-4ec2-416b-a66b-8b217991c918 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:10.482752 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e0fe81-4ec2-416b-a66b-8b217991c918 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:10.483233 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e0fe81-4ec2-416b-a66b-8b217991c918 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:10.503150 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-22e0fe81-4ec2-416b-a66b-8b217991c918 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:32:10.504736 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-22e0fe81-4ec2-416b-a66b-8b217991c918 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:32:10.509666 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbae63cb-de71-44eb-b527-a51d1c3770d0 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:10.510254 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbae63cb-de71-44eb-b527-a51d1c3770d0 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:10.510772 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbae63cb-de71-44eb-b527-a51d1c3770d0 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:10.512543 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 434/883] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:32:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:32:10.519940 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1b6651-865c-42cb-a056-70144203970d None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:10.520537 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1b6651-865c-42cb-a056-70144203970d None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:10.521038 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d1b6651-865c-42cb-a056-70144203970d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:10.536835 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:10.542055 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-7d1b6651-865c-42cb-a056-70144203970d None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:32:10.543623 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-7d1b6651-865c-42cb-a056-70144203970d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:32:10.598865 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 450/884] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:32:10 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:32:10.639165 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:10.643561 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 435/885] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:32:10 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:32:10.888899 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e0fe81-4ec2-416b-a66b-8b217991c918 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:10.918848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1b6651-865c-42cb-a056-70144203970d None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:10.999249 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e0fe81-4ec2-416b-a66b-8b217991c918 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:11.036307 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d1b6651-865c-42cb-a056-70144203970d None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:11.038907 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6535cf93-5f6a-4b79-b594-66bcec9a9f9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:11.039754 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6535cf93-5f6a-4b79-b594-66bcec9a9f9a None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:11.040194 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6535cf93-5f6a-4b79-b594-66bcec9a9f9a None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:11.040606 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6535cf93-5f6a-4b79-b594-66bcec9a9f9a None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:11.052232 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6535cf93-5f6a-4b79-b594-66bcec9a9f9a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:32:11.058366 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-6535cf93-5f6a-4b79-b594-66bcec9a9f9a None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 28 09:32:11.058366 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:11.058366 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:11.058366 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:11.058366 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:11.058366 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:11.058366 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:11.058366 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:11.058366 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:11.058366 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:11.058366 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:11.058366 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:11.058366 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:32:11.058366 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:32:11.058366 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:32:11.058366 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:32:11.059532 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:32:11.059532 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:32:11.059532 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:32:11.059532 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:32:11.059532 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:11.059532 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:11.059532 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:11.059532 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:11.059532 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:11.059532 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:11.059532 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:11.059532 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:11.059532 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:11.059532 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:11.059532 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:11.059532 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:11.060545 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:11.060545 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:11.060545 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:11.060545 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:11.060545 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:32:11.060545 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:32:11.060545 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:32:11.060545 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:32:11.060545 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:32:11.060545 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:32:11.060545 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 28 09:32:11.060545 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:11.063028 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 451/886] 10.222.0.25 () {56 vars in 1120 bytes} [Fri Oct 28 09:32:10 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:11.074471 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6535cf93-5f6a-4b79-b594-66bcec9a9f9a None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:11.077972 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43382d00-0a98-4261-8338-5d34dd75c36a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:11.079845 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43382d00-0a98-4261-8338-5d34dd75c36a None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:11.080353 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43382d00-0a98-4261-8338-5d34dd75c36a None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:11.080830 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43382d00-0a98-4261-8338-5d34dd75c36a None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:11.081633 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43382d00-0a98-4261-8338-5d34dd75c36a None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:11.083293 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43382d00-0a98-4261-8338-5d34dd75c36a None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:11.098635 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-43382d00-0a98-4261-8338-5d34dd75c36a None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 28 09:32:11.098635 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:11.098635 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 09:32:11.098635 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 09:32:11.098635 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 09:32:11.098635 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 09:32:11.098635 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 09:32:11.098635 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:32:11.098635 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 28 09:32:11.098635 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:11.098635 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 09:32:11.098635 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:11.098635 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:11.098635 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:11.098635 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:11.098635 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:11.098635 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:11.099769 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:11.099769 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:11.099769 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:11.099769 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:11.099769 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:11.099769 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:11.099769 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 09:32:11.099769 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 09:32:11.099769 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 09:32:11.099769 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 09:32:11.099769 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:11.099769 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:11.099769 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:11.099769 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:11.099769 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:11.099769 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:11.100699 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:11.100699 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:11.100699 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:11.100699 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:11.100699 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:11.100699 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:11.100699 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:11.100699 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:11.100699 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:11.100699 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:11.100699 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 09:32:11.100699 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:32:11.100699 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 28 09:32:11.100699 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:11.108611 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 436/887] 10.222.0.25 () {56 vars in 1114 bytes} [Fri Oct 28 09:32:10 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:32:11.110010 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:11.110328 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43382d00-0a98-4261-8338-5d34dd75c36a None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:11.198270 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:11.214708 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6535cf93-5f6a-4b79-b594-66bcec9a9f9a None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:11.256571 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43382d00-0a98-4261-8338-5d34dd75c36a None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:11.264443 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acae1604-11cb-4261-b989-26afc40031e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:11.266057 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acae1604-11cb-4261-b989-26afc40031e1 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:11.266489 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acae1604-11cb-4261-b989-26afc40031e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:11.266866 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acae1604-11cb-4261-b989-26afc40031e1 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:11.267672 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acae1604-11cb-4261-b989-26afc40031e1 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:11.268083 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acae1604-11cb-4261-b989-26afc40031e1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:11.270165 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acae1604-11cb-4261-b989-26afc40031e1 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:11.287313 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 452/888] 10.222.0.25 () {56 vars in 1115 bytes} [Fri Oct 28 09:32:11 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:11.288857 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acae1604-11cb-4261-b989-26afc40031e1 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:11.306944 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076217e0-9c6c-4453-8498-9c663c3b122f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:11.307575 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076217e0-9c6c-4453-8498-9c663c3b122f None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:11.308012 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076217e0-9c6c-4453-8498-9c663c3b122f None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:11.308306 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076217e0-9c6c-4453-8498-9c663c3b122f None admin] PATH_INFO: `/v3/services` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:11.309278 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-076217e0-9c6c-4453-8498-9c663c3b122f None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:11.309590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-076217e0-9c6c-4453-8498-9c663c3b122f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:11.311025 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-076217e0-9c6c-4453-8498-9c663c3b122f None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:11.323108 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 437/889] 10.222.0.25 () {56 vars in 1114 bytes} [Fri Oct 28 09:32:11 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:11.324743 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b7810b-6b76-416b-ac34-633be678ef23 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:11.324933 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b7810b-6b76-416b-ac34-633be678ef23 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:11.325220 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38b7810b-6b76-416b-ac34-633be678ef23 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:11.326651 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 438/890] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:32:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:32:11.327264 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b7810b-6b76-416b-ac34-633be678ef23 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:11.394199 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:11.412427 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acae1604-11cb-4261-b989-26afc40031e1 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:11.441488 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38b7810b-6b76-416b-ac34-633be678ef23 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:11.450893 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2a1a620-617e-4b3b-a182-f81854081c4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:11.452002 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a1a620-617e-4b3b-a182-f81854081c4f None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:11.452292 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a1a620-617e-4b3b-a182-f81854081c4f None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:11.452575 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a1a620-617e-4b3b-a182-f81854081c4f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:11.464561 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:11.464754 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a1a620-617e-4b3b-a182-f81854081c4f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:11.466316 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2a1a620-617e-4b3b-a182-f81854081c4f None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:11.480419 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c002d0b-7f26-4d51-85e7-a62f20d1c036 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:11.481107 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c002d0b-7f26-4d51-85e7-a62f20d1c036 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:11.481272 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c002d0b-7f26-4d51-85e7-a62f20d1c036 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:11.481551 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c002d0b-7f26-4d51-85e7-a62f20d1c036 None admin] PATH_INFO: `/v3/services` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:11.482186 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c002d0b-7f26-4d51-85e7-a62f20d1c036 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:11.482529 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c002d0b-7f26-4d51-85e7-a62f20d1c036 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:11.483365 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-c2a1a620-617e-4b3b-a182-f81854081c4f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:32:11.483365 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:11.483365 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:11.483365 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:11.483365 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:11.483365 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:11.483365 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:11.483365 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:11.483365 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:11.483365 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:11.483365 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:11.483365 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:11.483365 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:32:11.483365 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:32:11.483365 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:32:11.483365 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:32:11.484612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:11.484612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:11.484612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:11.484612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:11.484612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:11.484612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:11.484612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:11.484612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:11.484612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:11.484612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:11.484612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:11.484612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:11.484612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:11.484612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:11.484612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:11.484612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:11.484612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:32:11.491199 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:32:11.491199 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:32:11.491199 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:11.491389 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c002d0b-7f26-4d51-85e7-a62f20d1c036 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:11.491466 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 453/891] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:32:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:11.491466 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334916ab-e5fd-4bc9-8b2d-541771688ae6 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:11.491466 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334916ab-e5fd-4bc9-8b2d-541771688ae6 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:11.491466 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334916ab-e5fd-4bc9-8b2d-541771688ae6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:11.495629 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 439/892] 10.222.0.25 () {56 vars in 1114 bytes} [Fri Oct 28 09:32:11 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:11.496343 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c002d0b-7f26-4d51-85e7-a62f20d1c036 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:11.514741 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-334916ab-e5fd-4bc9-8b2d-541771688ae6 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:32:11.515949 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-334916ab-e5fd-4bc9-8b2d-541771688ae6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:32:11.598903 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 454/893] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:32:11 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:32:11.601091 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-334916ab-e5fd-4bc9-8b2d-541771688ae6 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:11.615048 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c002d0b-7f26-4d51-85e7-a62f20d1c036 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:11.652219 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f830bad2-30d0-422a-99ed-6c0f79ff4105 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:11.653204 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f830bad2-30d0-422a-99ed-6c0f79ff4105 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:11.653707 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f830bad2-30d0-422a-99ed-6c0f79ff4105 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:11.654193 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f830bad2-30d0-422a-99ed-6c0f79ff4105 None admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:11.655084 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f830bad2-30d0-422a-99ed-6c0f79ff4105 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:11.655608 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f830bad2-30d0-422a-99ed-6c0f79ff4105 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:11.657178 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f830bad2-30d0-422a-99ed-6c0f79ff4105 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:11.671606 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 440/894] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:32:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:11.676787 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f830bad2-30d0-422a-99ed-6c0f79ff4105 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:11.691131 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:11.729269 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-334916ab-e5fd-4bc9-8b2d-541771688ae6 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:11.740203 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:11.766911 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0edc86a-2240-47cd-9e0b-6b5656f2a534 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:11.767599 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0edc86a-2240-47cd-9e0b-6b5656f2a534 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:11.767895 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0edc86a-2240-47cd-9e0b-6b5656f2a534 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:11.768190 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0edc86a-2240-47cd-9e0b-6b5656f2a534 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:11.768970 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0edc86a-2240-47cd-9e0b-6b5656f2a534 None admin] RBAC: Adding query filter params (interface=public, service_id=9b902d226c0141fdbf22f475ba642a6f, region_id=RegionOne) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:11.769292 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0edc86a-2240-47cd-9e0b-6b5656f2a534 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:11.771426 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0edc86a-2240-47cd-9e0b-6b5656f2a534 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:11.784189 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 455/895] 10.222.0.25 () {56 vars in 1251 bytes} [Fri Oct 28 09:32:11 2022] GET /identity/v3/endpoints?service_id=9b902d226c0141fdbf22f475ba642a6f&interface=public®ion_id=RegionOne => generated 194 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:11.785510 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0edc86a-2240-47cd-9e0b-6b5656f2a534 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:11.826888 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f830bad2-30d0-422a-99ed-6c0f79ff4105 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:11.868450 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf564379-9532-4abf-b9d7-9c2fabe407b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:11.869775 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf564379-9532-4abf-b9d7-9c2fabe407b6 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:11.870053 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf564379-9532-4abf-b9d7-9c2fabe407b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:11.870330 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf564379-9532-4abf-b9d7-9c2fabe407b6 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:11.880815 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf564379-9532-4abf-b9d7-9c2fabe407b6 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:11.883332 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf564379-9532-4abf-b9d7-9c2fabe407b6 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:11.899198 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-cf564379-9532-4abf-b9d7-9c2fabe407b6 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 28 09:32:11.899198 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:11.899198 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:11.899198 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:11.899198 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:11.899198 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:11.899198 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:11.899198 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:11.899198 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:11.899198 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:11.899198 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:11.899198 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:11.899198 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 09:32:11.899198 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 09:32:11.899198 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 09:32:11.899198 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 09:32:11.900208 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:11.900208 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:11.900208 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 09:32:11.900208 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:32:11.900208 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 09:32:11.900208 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:32:11.900208 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:11.900208 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:11.900208 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:11.900208 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:11.900208 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:11.900208 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:11.900208 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:11.900208 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:11.900208 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:11.900208 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:11.900208 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:11.900208 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:11.901169 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:11.901169 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:11.901169 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 09:32:11.901169 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 09:32:11.901169 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 09:32:11.901169 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 09:32:11.901169 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 09:32:11.901169 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 09:32:11.901169 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 28 09:32:11.901169 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:11.902238 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 441/896] 10.222.0.25 () {56 vars in 1117 bytes} [Fri Oct 28 09:32:11 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:32:11.903080 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf564379-9532-4abf-b9d7-9c2fabe407b6 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:11.913256 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0edc86a-2240-47cd-9e0b-6b5656f2a534 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:11.943032 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:11.949831 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb76d27-a02b-4938-bfe1-24e1f84aa9a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:11.950478 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb76d27-a02b-4938-bfe1-24e1f84aa9a3 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:11.950762 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb76d27-a02b-4938-bfe1-24e1f84aa9a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:11.951215 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb76d27-a02b-4938-bfe1-24e1f84aa9a3 None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:11.951845 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb76d27-a02b-4938-bfe1-24e1f84aa9a3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:11.953203 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb76d27-a02b-4938-bfe1-24e1f84aa9a3 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:11.959128 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 456/897] 10.222.0.25 () {56 vars in 1087 bytes} [Fri Oct 28 09:32:11 2022] GET /identity/v3/services => generated 1268 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:32:11.960485 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7761b6-434e-43c7-bd7e-5e29de1ca96d None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:11.960780 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7761b6-434e-43c7-bd7e-5e29de1ca96d None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:11.961061 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7761b6-434e-43c7-bd7e-5e29de1ca96d None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:11.962453 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 457/898] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:32:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:32:11.967533 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7761b6-434e-43c7-bd7e-5e29de1ca96d None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:12.019684 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf564379-9532-4abf-b9d7-9c2fabe407b6 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:12.023906 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:12.060893 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7327280f-8c03-42ed-b679-041d24867fbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:12.061552 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7327280f-8c03-42ed-b679-041d24867fbe None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:12.061842 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7327280f-8c03-42ed-b679-041d24867fbe None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:12.062119 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7327280f-8c03-42ed-b679-041d24867fbe None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:12.062751 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7327280f-8c03-42ed-b679-041d24867fbe None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:12.064290 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7327280f-8c03-42ed-b679-041d24867fbe None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:12.074302 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-7327280f-8c03-42ed-b679-041d24867fbe None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 28 09:32:12.074302 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:12.074302 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 09:32:12.074302 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 09:32:12.074302 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 09:32:12.074302 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 09:32:12.074302 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 09:32:12.074302 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:32:12.074302 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 28 09:32:12.074302 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:12.074302 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 09:32:12.074302 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:12.074302 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:12.074302 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:12.074302 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:12.074302 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:12.075623 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:12.075623 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:12.075623 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:12.075623 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:12.075623 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:12.075623 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:12.075623 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:12.075623 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 09:32:12.075623 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 09:32:12.075623 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 09:32:12.075623 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 09:32:12.075623 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:12.075623 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:12.075623 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:12.075623 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:12.075623 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:12.075623 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:12.076569 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:12.076569 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:12.076569 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:12.076569 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:12.076569 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:12.076569 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:12.076569 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:12.076569 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:12.076569 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:12.076569 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:12.076569 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 09:32:12.076569 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:32:12.076569 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 28 09:32:12.076569 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:12.076569 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 442/899] 10.222.0.25 () {56 vars in 1132 bytes} [Fri Oct 28 09:32:11 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:12.077968 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e91d53f-06a3-443a-bbb8-bb6b75164ea4 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:12.077968 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e91d53f-06a3-443a-bbb8-bb6b75164ea4 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:12.077968 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e91d53f-06a3-443a-bbb8-bb6b75164ea4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:12.098173 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-9e91d53f-06a3-443a-bbb8-bb6b75164ea4 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:32:12.099460 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-9e91d53f-06a3-443a-bbb8-bb6b75164ea4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:32:12.106033 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7761b6-434e-43c7-bd7e-5e29de1ca96d None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:12.150498 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7003dd-91a5-4d7d-a5f1-439a68ce2aa5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:12.151132 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7003dd-91a5-4d7d-a5f1-439a68ce2aa5 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:12.151409 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7003dd-91a5-4d7d-a5f1-439a68ce2aa5 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:12.151688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7003dd-91a5-4d7d-a5f1-439a68ce2aa5 None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:12.152441 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7003dd-91a5-4d7d-a5f1-439a68ce2aa5 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:12.152748 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7003dd-91a5-4d7d-a5f1-439a68ce2aa5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:12.154204 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7003dd-91a5-4d7d-a5f1-439a68ce2aa5 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:12.179722 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 458/900] 10.222.0.25 () {56 vars in 1149 bytes} [Fri Oct 28 09:32:11 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:12.180610 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7003dd-91a5-4d7d-a5f1-439a68ce2aa5 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:12.196145 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 443/901] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:32:12 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:32:12.197280 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e91d53f-06a3-443a-bbb8-bb6b75164ea4 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:12.260895 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:12.327565 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7003dd-91a5-4d7d-a5f1-439a68ce2aa5 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:12.354522 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:12.361047 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e91d53f-06a3-443a-bbb8-bb6b75164ea4 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:12.374553 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9cfb86-acef-40af-af5b-db19be81968f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:12.375319 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9cfb86-acef-40af-af5b-db19be81968f None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:12.376151 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9cfb86-acef-40af-af5b-db19be81968f None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:12.376151 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9cfb86-acef-40af-af5b-db19be81968f None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:12.376800 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da9cfb86-acef-40af-af5b-db19be81968f None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:12.377335 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da9cfb86-acef-40af-af5b-db19be81968f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:12.378787 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da9cfb86-acef-40af-af5b-db19be81968f None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:12.385178 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 459/902] 10.222.0.25 () {56 vars in 1126 bytes} [Fri Oct 28 09:32:12 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:12.390428 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9cfb86-acef-40af-af5b-db19be81968f None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:12.422672 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940bbeac-d3fc-4fe1-80ba-af28b67af501 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:12.423945 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940bbeac-d3fc-4fe1-80ba-af28b67af501 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:12.424943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940bbeac-d3fc-4fe1-80ba-af28b67af501 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:12.425675 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940bbeac-d3fc-4fe1-80ba-af28b67af501 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:12.511651 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-940bbeac-d3fc-4fe1-80ba-af28b67af501 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:12.513170 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-940bbeac-d3fc-4fe1-80ba-af28b67af501 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:12.529754 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9cfb86-acef-40af-af5b-db19be81968f None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:12.552451 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:12.581036 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70d2a40-1752-4c79-8274-35f010babef7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:12.581665 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70d2a40-1752-4c79-8274-35f010babef7 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:12.581951 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70d2a40-1752-4c79-8274-35f010babef7 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:12.582227 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70d2a40-1752-4c79-8274-35f010babef7 None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:12.582929 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d70d2a40-1752-4c79-8274-35f010babef7 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:12.583239 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d70d2a40-1752-4c79-8274-35f010babef7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:12.584671 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d70d2a40-1752-4c79-8274-35f010babef7 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:12.590423 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 460/903] 10.222.0.25 () {56 vars in 1126 bytes} [Fri Oct 28 09:32:12 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:32:12.592243 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70d2a40-1752-4c79-8274-35f010babef7 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:12.631585 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:12.637212 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 444/904] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:32:12 2022] GET /identity/v3/auth/tokens => generated 1372 bytes in 440 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 09:32:12.638590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940bbeac-d3fc-4fe1-80ba-af28b67af501 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:12.714664 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70d2a40-1752-4c79-8274-35f010babef7 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:12.754873 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90dbc697-2970-488a-9dd3-a6da12e2b7c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:12.755501 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90dbc697-2970-488a-9dd3-a6da12e2b7c7 None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:12.755862 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90dbc697-2970-488a-9dd3-a6da12e2b7c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:12.756067 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90dbc697-2970-488a-9dd3-a6da12e2b7c7 None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:12.756784 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90dbc697-2970-488a-9dd3-a6da12e2b7c7 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:12.758174 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90dbc697-2970-488a-9dd3-a6da12e2b7c7 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:12.758427 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940bbeac-d3fc-4fe1-80ba-af28b67af501 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:12.775891 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 461/905] 10.222.0.25 () {60 vars in 1141 bytes} [Fri Oct 28 09:32:12 2022] POST /identity/v3/services => generated 241 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:32:12.776555 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90dbc697-2970-488a-9dd3-a6da12e2b7c7 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:12.825898 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755569e1-e9ad-459d-93d5-1531f6c0672e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:12.826559 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755569e1-e9ad-459d-93d5-1531f6c0672e None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:12.826802 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755569e1-e9ad-459d-93d5-1531f6c0672e None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:12.827077 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755569e1-e9ad-459d-93d5-1531f6c0672e None admin] PATH_INFO: `/v3/endpoints` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:12.827801 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-755569e1-e9ad-459d-93d5-1531f6c0672e None admin] RBAC: Adding query filter params (interface=public, service_id=1b5d8a8889054c3da3875ce7cf366eda, region_id=RegionOne) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:12.828116 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-755569e1-e9ad-459d-93d5-1531f6c0672e None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:12.829586 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-755569e1-e9ad-459d-93d5-1531f6c0672e None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:12.836462 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 445/906] 10.222.0.25 () {56 vars in 1251 bytes} [Fri Oct 28 09:32:12 2022] GET /identity/v3/endpoints?service_id=1b5d8a8889054c3da3875ce7cf366eda&interface=public®ion_id=RegionOne => generated 194 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:12.842492 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755569e1-e9ad-459d-93d5-1531f6c0672e None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:12.850372 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:12.919824 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:12.942929 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90dbc697-2970-488a-9dd3-a6da12e2b7c7 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:12.986030 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755569e1-e9ad-459d-93d5-1531f6c0672e None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:13.001340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-710e2bf4-c326-40ba-bea2-354e9db90217 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:13.002945 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710e2bf4-c326-40ba-bea2-354e9db90217 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:13.002945 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710e2bf4-c326-40ba-bea2-354e9db90217 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:13.002945 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710e2bf4-c326-40ba-bea2-354e9db90217 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:13.019661 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-710e2bf4-c326-40ba-bea2-354e9db90217 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:13.021345 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-710e2bf4-c326-40ba-bea2-354e9db90217 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:13.035207 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-990a6000-05cc-4fd5-808e-3143b3507613 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:13.036114 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990a6000-05cc-4fd5-808e-3143b3507613 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:13.036535 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990a6000-05cc-4fd5-808e-3143b3507613 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:13.037035 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-990a6000-05cc-4fd5-808e-3143b3507613 None admin] PATH_INFO: `/v3/services` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:13.037184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-710e2bf4-c326-40ba-bea2-354e9db90217 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:32:13.037184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:13.037184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:13.037184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:13.037184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:13.037184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:13.037184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:13.037184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:13.037184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:13.037184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:13.037184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:13.037184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:13.037184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:32:13.037184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:32:13.037184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:32:13.037184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:32:13.038654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:13.038654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:13.038654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:13.038654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:13.038654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:13.038654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:13.038654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:13.038654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:13.038654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:13.038654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:13.038654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:13.038654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:13.038654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:13.038654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:13.038654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:13.038654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:13.038654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:32:13.039694 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:32:13.039694 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:32:13.039694 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:13.039694 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 462/907] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:32:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 263 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:13.040684 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-990a6000-05cc-4fd5-808e-3143b3507613 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:13.042388 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-990a6000-05cc-4fd5-808e-3143b3507613 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:13.045142 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-710e2bf4-c326-40ba-bea2-354e9db90217 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:13.049632 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 446/908] 10.222.0.25 () {56 vars in 1087 bytes} [Fri Oct 28 09:32:12 2022] GET /identity/v3/services => generated 1497 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:32:13.174594 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:13.183335 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-710e2bf4-c326-40ba-bea2-354e9db90217 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:13.229398 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ca3c2d-af80-486e-b28a-b7c73f103b06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:13.230193 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ca3c2d-af80-486e-b28a-b7c73f103b06 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:13.230328 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ca3c2d-af80-486e-b28a-b7c73f103b06 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:13.230625 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3ca3c2d-af80-486e-b28a-b7c73f103b06 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:13.231312 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3ca3c2d-af80-486e-b28a-b7c73f103b06 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:13.231621 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3ca3c2d-af80-486e-b28a-b7c73f103b06 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:13.233058 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3ca3c2d-af80-486e-b28a-b7c73f103b06 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:13.248657 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 463/909] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:32:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:13.254527 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-990a6000-05cc-4fd5-808e-3143b3507613 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:13.375577 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-990a6000-05cc-4fd5-808e-3143b3507613 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:13.405269 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:13.419246 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5794680-f835-458d-b5c0-c3d933493f0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:13.420126 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5794680-f835-458d-b5c0-c3d933493f0d None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:13.420560 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5794680-f835-458d-b5c0-c3d933493f0d None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:13.421006 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5794680-f835-458d-b5c0-c3d933493f0d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:13.432654 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5794680-f835-458d-b5c0-c3d933493f0d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:13.434349 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5794680-f835-458d-b5c0-c3d933493f0d None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:13.447569 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-c5794680-f835-458d-b5c0-c3d933493f0d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:32:13.447569 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:13.447569 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:13.447569 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:13.447569 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:13.447569 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:13.447569 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:13.447569 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:13.447569 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:13.447569 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:13.447569 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:13.447569 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:13.447569 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 09:32:13.447569 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 09:32:13.447569 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 09:32:13.448681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 09:32:13.448681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:13.448681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:13.448681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:13.448681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:13.448681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:13.448681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:13.448681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:13.448681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:13.448681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:13.448681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:13.448681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:13.448681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:13.448681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:13.448681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:13.448681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:13.448681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:13.449859 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 09:32:13.449859 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 09:32:13.449859 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 09:32:13.449859 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 09:32:13.449859 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 09:32:13.449859 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 09:32:13.449859 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:32:13.449859 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:13.451993 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 447/910] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:32:13 2022] GET /identity/v3/projects/service => generated 88 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:13.462462 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ca3c2d-af80-486e-b28a-b7c73f103b06 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:13.568959 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3ca3c2d-af80-486e-b28a-b7c73f103b06 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:13.608011 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b840b2-dfc1-4bc1-8fb6-239a232c1e13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:13.608631 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b840b2-dfc1-4bc1-8fb6-239a232c1e13 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:13.608982 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b840b2-dfc1-4bc1-8fb6-239a232c1e13 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:13.609274 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b840b2-dfc1-4bc1-8fb6-239a232c1e13 None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:13.609972 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b840b2-dfc1-4bc1-8fb6-239a232c1e13 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:13.610281 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b840b2-dfc1-4bc1-8fb6-239a232c1e13 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:13.611726 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9b840b2-dfc1-4bc1-8fb6-239a232c1e13 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:13.624938 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 464/911] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Oct 28 09:32:13 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:13.632532 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5794680-f835-458d-b5c0-c3d933493f0d None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:13.776947 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5794680-f835-458d-b5c0-c3d933493f0d None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:13.818451 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d72faaa6-be88-4ccf-8863-317286906916 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:13.820032 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72faaa6-be88-4ccf-8863-317286906916 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:13.820564 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72faaa6-be88-4ccf-8863-317286906916 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:13.821047 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72faaa6-be88-4ccf-8863-317286906916 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:13.822007 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d72faaa6-be88-4ccf-8863-317286906916 None admin] RBAC: Adding query filter params (role.id=4e19a68eecb04cb998e6aa7adfb48ad3, scope.project.id=455d1c826b974bc38497d18a516867c7, user.id=97c8877c74194a6bbbf25b3c8f5b7dbd) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:13.822573 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d72faaa6-be88-4ccf-8863-317286906916 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:13.824247 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d72faaa6-be88-4ccf-8863-317286906916 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:13.830719 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 448/912] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:32:13 2022] GET /identity/v3/role_assignments?user.id=97c8877c74194a6bbbf25b3c8f5b7dbd&scope.project.id=455d1c826b974bc38497d18a516867c7&role.id=4e19a68eecb04cb998e6aa7adfb48ad3 => generated 259 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:14.255407 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8146e3-a35b-4353-a0e8-9b2df16eaf64 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:14.255824 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8146e3-a35b-4353-a0e8-9b2df16eaf64 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:14.256010 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8146e3-a35b-4353-a0e8-9b2df16eaf64 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:14.257670 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 465/913] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:32:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:32:14.263261 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2812983-fcd5-4a2f-9e9c-3c0303acac3b None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:14.263550 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2812983-fcd5-4a2f-9e9c-3c0303acac3b None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:14.263831 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2812983-fcd5-4a2f-9e9c-3c0303acac3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:14.284008 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-f2812983-fcd5-4a2f-9e9c-3c0303acac3b None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:32:14.285208 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-f2812983-fcd5-4a2f-9e9c-3c0303acac3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:32:14.332377 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:14.360699 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 449/914] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:32:14 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 09:32:14.640859 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de8146e3-a35b-4353-a0e8-9b2df16eaf64 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:14.662251 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:14.757316 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de8146e3-a35b-4353-a0e8-9b2df16eaf64 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:14.794414 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478f0742-6b87-49e9-9c3f-1d1fc53659a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:14.795057 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478f0742-6b87-49e9-9c3f-1d1fc53659a9 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:14.795334 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478f0742-6b87-49e9-9c3f-1d1fc53659a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:14.795600 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478f0742-6b87-49e9-9c3f-1d1fc53659a9 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:14.796177 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-478f0742-6b87-49e9-9c3f-1d1fc53659a9 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:14.797531 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-478f0742-6b87-49e9-9c3f-1d1fc53659a9 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:14.807186 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-478f0742-6b87-49e9-9c3f-1d1fc53659a9 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 28 09:32:14.807186 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:14.807186 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 09:32:14.807186 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 09:32:14.807186 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 09:32:14.807186 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 09:32:14.807186 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 09:32:14.807186 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:32:14.807186 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 28 09:32:14.807186 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:14.807186 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 09:32:14.807186 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:14.807186 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:14.807186 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:14.807186 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:14.807186 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:14.807186 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:14.808236 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:14.808236 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:14.808236 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:14.808236 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:14.808236 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:14.808236 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:14.808236 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 09:32:14.808236 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 09:32:14.808236 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 09:32:14.808236 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 09:32:14.808236 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:14.808236 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:14.808236 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:14.808236 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:14.808236 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:14.808236 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:14.809128 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:14.809128 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:14.809128 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:14.809128 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:14.809128 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:14.809128 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:14.809128 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:14.809128 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:14.809128 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:14.809128 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:14.809128 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 09:32:14.809128 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:32:14.809128 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Oct 28 09:32:14.809128 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:14.809128 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 466/915] 10.222.0.25 () {56 vars in 1114 bytes} [Fri Oct 28 09:32:14 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:14.814233 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2812983-fcd5-4a2f-9e9c-3c0303acac3b None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:14.923832 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2812983-fcd5-4a2f-9e9c-3c0303acac3b None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:14.960192 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-291b5aaf-fa77-412f-bfb1-8171aa964547 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:14.961184 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291b5aaf-fa77-412f-bfb1-8171aa964547 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:14.961673 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291b5aaf-fa77-412f-bfb1-8171aa964547 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:14.962192 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291b5aaf-fa77-412f-bfb1-8171aa964547 None admin] PATH_INFO: `/v3/services` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:14.963052 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-291b5aaf-fa77-412f-bfb1-8171aa964547 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:14.963598 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-291b5aaf-fa77-412f-bfb1-8171aa964547 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:14.965170 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-291b5aaf-fa77-412f-bfb1-8171aa964547 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:14.970581 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 450/916] 10.222.0.25 () {56 vars in 1114 bytes} [Fri Oct 28 09:32:14 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:14.978440 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478f0742-6b87-49e9-9c3f-1d1fc53659a9 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:15.057710 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:15.079825 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478f0742-6b87-49e9-9c3f-1d1fc53659a9 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:15.117666 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dbf4456-bdc6-4678-836d-d964f3046b27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:15.118351 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbf4456-bdc6-4678-836d-d964f3046b27 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:15.118666 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbf4456-bdc6-4678-836d-d964f3046b27 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:15.118943 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbf4456-bdc6-4678-836d-d964f3046b27 None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:15.119593 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dbf4456-bdc6-4678-836d-d964f3046b27 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:15.119899 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dbf4456-bdc6-4678-836d-d964f3046b27 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:15.121293 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dbf4456-bdc6-4678-836d-d964f3046b27 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:15.127488 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 467/917] 10.222.0.25 () {56 vars in 1114 bytes} [Fri Oct 28 09:32:14 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:15.134750 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-291b5aaf-fa77-412f-bfb1-8171aa964547 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:15.213208 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:15.245940 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc953a2-3b70-4df1-9965-10e3e011c69b None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:15.246515 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc953a2-3b70-4df1-9965-10e3e011c69b None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:15.246885 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc953a2-3b70-4df1-9965-10e3e011c69b None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:15.249450 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 468/918] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:32:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 09:32:15.255295 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e475357-5815-46c0-b7d8-1b6ed075f271 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:15.255656 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e475357-5815-46c0-b7d8-1b6ed075f271 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:15.256041 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e475357-5815-46c0-b7d8-1b6ed075f271 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:15.278588 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-3e475357-5815-46c0-b7d8-1b6ed075f271 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:32:15.280560 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-3e475357-5815-46c0-b7d8-1b6ed075f271 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:32:15.282197 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-291b5aaf-fa77-412f-bfb1-8171aa964547 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:15.321267 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fccb041-7085-46cd-a1d8-17e09b7f7730 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:15.322231 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fccb041-7085-46cd-a1d8-17e09b7f7730 None admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:15.322751 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fccb041-7085-46cd-a1d8-17e09b7f7730 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:15.323417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fccb041-7085-46cd-a1d8-17e09b7f7730 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:15.324496 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fccb041-7085-46cd-a1d8-17e09b7f7730 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:15.326164 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fccb041-7085-46cd-a1d8-17e09b7f7730 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:15.352175 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 451/919] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:32:15 2022] POST /identity/v3/endpoints => generated 346 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:32:15.378017 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 469/920] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:32:15 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:32:15.658917 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fccb041-7085-46cd-a1d8-17e09b7f7730 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:15.682411 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf75259-9052-41b1-b7d6-e65c0f6405d7 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:15.682951 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf75259-9052-41b1-b7d6-e65c0f6405d7 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:15.683030 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf75259-9052-41b1-b7d6-e65c0f6405d7 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:15.684626 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 470/921] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:32:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 09:32:15.690438 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf451d6-282e-4145-99e4-7b8cdf58a3a0 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:15.690719 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf451d6-282e-4145-99e4-7b8cdf58a3a0 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:15.691212 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf451d6-282e-4145-99e4-7b8cdf58a3a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:15.709931 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-edf451d6-282e-4145-99e4-7b8cdf58a3a0 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:32:15.711145 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-edf451d6-282e-4145-99e4-7b8cdf58a3a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:32:15.779727 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:15.786136 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:15.787444 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fccb041-7085-46cd-a1d8-17e09b7f7730 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:15.802591 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 471/922] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:32:15 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 09:32:15.826467 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3391b8-c65a-4b1f-82ed-4d8c7f02a34d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:15.827771 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3391b8-c65a-4b1f-82ed-4d8c7f02a34d None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:15.828250 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3391b8-c65a-4b1f-82ed-4d8c7f02a34d None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:15.828740 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3391b8-c65a-4b1f-82ed-4d8c7f02a34d None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:15.829558 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e3391b8-c65a-4b1f-82ed-4d8c7f02a34d None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:15.831151 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e3391b8-c65a-4b1f-82ed-4d8c7f02a34d None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:15.841774 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-7e3391b8-c65a-4b1f-82ed-4d8c7f02a34d None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 09:32:15.841774 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:15.841774 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 09:32:15.841774 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 09:32:15.841774 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 09:32:15.841774 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 09:32:15.841774 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 09:32:15.841774 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:32:15.841774 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 09:32:15.841774 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:15.841774 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 09:32:15.841774 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:15.841774 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:15.841774 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:15.841774 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:15.841774 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:15.841774 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:15.842819 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:15.842819 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:15.842819 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:15.842819 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:15.842819 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:15.842819 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:15.842819 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 09:32:15.842819 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 09:32:15.842819 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 09:32:15.842819 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 09:32:15.842819 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:15.842819 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:15.842819 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:15.842819 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:15.842819 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:15.842819 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:15.843653 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:15.843653 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:15.843653 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:15.843653 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:15.843653 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:15.843653 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:15.843653 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:15.843653 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:15.843653 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:15.843653 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:15.843653 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 09:32:15.843653 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:32:15.843653 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 09:32:15.843653 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:15.846416 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 452/923] 10.222.0.25 () {56 vars in 1126 bytes} [Fri Oct 28 09:32:15 2022] GET /identity/v3/services/object-store => generated 93 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:15.858421 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf451d6-282e-4145-99e4-7b8cdf58a3a0 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:15.957912 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf451d6-282e-4145-99e4-7b8cdf58a3a0 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:15.994011 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130ebdca-b311-44b9-a62b-a561c1a7beba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:15.994728 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130ebdca-b311-44b9-a62b-a561c1a7beba None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:15.995020 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130ebdca-b311-44b9-a62b-a561c1a7beba None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:15.995300 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130ebdca-b311-44b9-a62b-a561c1a7beba None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:15.995954 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-130ebdca-b311-44b9-a62b-a561c1a7beba None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:15.996267 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-130ebdca-b311-44b9-a62b-a561c1a7beba None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:15.997630 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-130ebdca-b311-44b9-a62b-a561c1a7beba None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:16.003099 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 472/924] 10.222.0.25 () {56 vars in 1122 bytes} [Fri Oct 28 09:32:15 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:16.009402 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3391b8-c65a-4b1f-82ed-4d8c7f02a34d None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:16.106798 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130ebdca-b311-44b9-a62b-a561c1a7beba None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:16.126535 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3391b8-c65a-4b1f-82ed-4d8c7f02a34d None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:16.165907 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d2c2e2-5af4-4d83-b92f-6b1a32d5110e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:16.166560 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d2c2e2-5af4-4d83-b92f-6b1a32d5110e None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:16.166839 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d2c2e2-5af4-4d83-b92f-6b1a32d5110e None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:16.167212 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d2c2e2-5af4-4d83-b92f-6b1a32d5110e None admin] PATH_INFO: `/v3/services` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:16.167880 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d2c2e2-5af4-4d83-b92f-6b1a32d5110e None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:16.168181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d2c2e2-5af4-4d83-b92f-6b1a32d5110e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:16.169661 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d2c2e2-5af4-4d83-b92f-6b1a32d5110e None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:16.176402 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:16.176772 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 453/925] 10.222.0.25 () {56 vars in 1122 bytes} [Fri Oct 28 09:32:16 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:16.183994 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d2c2e2-5af4-4d83-b92f-6b1a32d5110e None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:16.221916 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:16.225940 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130ebdca-b311-44b9-a62b-a561c1a7beba None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:16.263583 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe46d46-f674-45be-a974-e011823b9d54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:16.264585 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe46d46-f674-45be-a974-e011823b9d54 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:16.265069 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe46d46-f674-45be-a974-e011823b9d54 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:16.265540 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe46d46-f674-45be-a974-e011823b9d54 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:16.266369 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe46d46-f674-45be-a974-e011823b9d54 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:16.267952 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe46d46-f674-45be-a974-e011823b9d54 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:16.278182 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-bfe46d46-f674-45be-a974-e011823b9d54 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 28 09:32:16.278182 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:16.278182 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 09:32:16.278182 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 09:32:16.278182 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 09:32:16.278182 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 09:32:16.278182 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 09:32:16.278182 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:32:16.278182 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 28 09:32:16.278182 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:16.278182 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 09:32:16.278182 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:16.278182 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:16.278182 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:16.278182 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:16.278182 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:16.279196 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:16.279196 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:16.279196 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:16.279196 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:16.279196 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:16.279196 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:16.279196 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:16.279196 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 09:32:16.279196 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 09:32:16.279196 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 09:32:16.279196 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 09:32:16.279196 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:16.279196 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:16.279196 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:16.279196 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:16.279196 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:16.279196 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:16.280161 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:16.280161 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:16.280161 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:16.280161 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:16.280161 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:16.280161 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:16.280161 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:16.280161 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:16.280161 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:16.280161 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:16.280161 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 09:32:16.280161 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:32:16.280161 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Oct 28 09:32:16.280161 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:16.283025 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 473/926] 10.222.0.25 () {56 vars in 1132 bytes} [Fri Oct 28 09:32:16 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:32:16.289064 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe46d46-f674-45be-a974-e011823b9d54 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:16.318220 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d2c2e2-5af4-4d83-b92f-6b1a32d5110e None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:16.360288 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b9f64c-5a07-49f1-8c69-c4d5bdda6862 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:16.361428 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b9f64c-5a07-49f1-8c69-c4d5bdda6862 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:16.361890 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b9f64c-5a07-49f1-8c69-c4d5bdda6862 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:16.362358 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b9f64c-5a07-49f1-8c69-c4d5bdda6862 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:16.363515 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80b9f64c-5a07-49f1-8c69-c4d5bdda6862 None admin] RBAC: Adding query filter params (interface=public, service_id=84ddeb9619a8482796ab05b4a3cce595, region_id=RegionOne) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:16.364007 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80b9f64c-5a07-49f1-8c69-c4d5bdda6862 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:16.366112 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80b9f64c-5a07-49f1-8c69-c4d5bdda6862 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:16.371929 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 454/927] 10.222.0.25 () {56 vars in 1251 bytes} [Fri Oct 28 09:32:16 2022] GET /identity/v3/endpoints?service_id=84ddeb9619a8482796ab05b4a3cce595&interface=public®ion_id=RegionOne => generated 194 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:16.373418 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60c3474-8c74-4e1b-8e2e-cdc2fd3ad745 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:16.373832 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60c3474-8c74-4e1b-8e2e-cdc2fd3ad745 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:16.374245 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60c3474-8c74-4e1b-8e2e-cdc2fd3ad745 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:16.375636 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 455/928] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:32:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:32:16.377335 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e60c3474-8c74-4e1b-8e2e-cdc2fd3ad745 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:16.411061 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe46d46-f674-45be-a974-e011823b9d54 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:16.438486 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:16.450055 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f5b029-dada-4907-a7b2-b0682516bb5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:16.450821 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f5b029-dada-4907-a7b2-b0682516bb5e None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:16.451104 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f5b029-dada-4907-a7b2-b0682516bb5e None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:16.451884 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f5b029-dada-4907-a7b2-b0682516bb5e None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:16.452119 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f5b029-dada-4907-a7b2-b0682516bb5e None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:16.452423 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f5b029-dada-4907-a7b2-b0682516bb5e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:16.453781 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f5b029-dada-4907-a7b2-b0682516bb5e None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:16.465034 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 474/929] 10.222.0.25 () {56 vars in 1126 bytes} [Fri Oct 28 09:32:16 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:16.465034 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67be64b-cd06-409b-a848-8e7bfcab76ee None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:16.465034 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67be64b-cd06-409b-a848-8e7bfcab76ee None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:16.465034 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67be64b-cd06-409b-a848-8e7bfcab76ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:16.479469 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:16.482613 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-a67be64b-cd06-409b-a848-8e7bfcab76ee None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:32:16.483854 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a67be64b-cd06-409b-a848-8e7bfcab76ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:32:16.494038 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e60c3474-8c74-4e1b-8e2e-cdc2fd3ad745 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:16.528987 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f1f39c-aa38-4847-9573-40d28d96bc46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:16.529837 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f1f39c-aa38-4847-9573-40d28d96bc46 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:16.529951 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f1f39c-aa38-4847-9573-40d28d96bc46 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:16.530134 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8f1f39c-aa38-4847-9573-40d28d96bc46 None admin] PATH_INFO: `/v3/services` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:16.530761 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8f1f39c-aa38-4847-9573-40d28d96bc46 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:16.532221 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8f1f39c-aa38-4847-9573-40d28d96bc46 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:16.538960 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 456/930] 10.222.0.25 () {56 vars in 1087 bytes} [Fri Oct 28 09:32:16 2022] GET /identity/v3/services => generated 1497 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:32:16.543483 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f1f39c-aa38-4847-9573-40d28d96bc46 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:16.566836 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 475/931] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:32:16 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:32:16.682075 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8f1f39c-aa38-4847-9573-40d28d96bc46 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:16.723646 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cac4a78-9961-4815-aec6-bf3054d14676 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:16.724661 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cac4a78-9961-4815-aec6-bf3054d14676 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:16.725694 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cac4a78-9961-4815-aec6-bf3054d14676 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:16.725974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cac4a78-9961-4815-aec6-bf3054d14676 None admin] PATH_INFO: `/v3/services` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:16.726657 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cac4a78-9961-4815-aec6-bf3054d14676 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:16.726974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cac4a78-9961-4815-aec6-bf3054d14676 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:16.728379 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cac4a78-9961-4815-aec6-bf3054d14676 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:16.733999 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 457/932] 10.222.0.25 () {56 vars in 1126 bytes} [Fri Oct 28 09:32:16 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:16.743535 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67be64b-cd06-409b-a848-8e7bfcab76ee None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:16.836376 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cac4a78-9961-4815-aec6-bf3054d14676 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:16.860148 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67be64b-cd06-409b-a848-8e7bfcab76ee None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:16.865940 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:16.886835 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:16.903692 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40dda274-5e24-4fb7-9cb0-1a03db7d812b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:16.904374 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40dda274-5e24-4fb7-9cb0-1a03db7d812b None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:16.904658 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40dda274-5e24-4fb7-9cb0-1a03db7d812b None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:16.904934 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40dda274-5e24-4fb7-9cb0-1a03db7d812b None admin] PATH_INFO: `/v3/endpoints` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:16.905652 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40dda274-5e24-4fb7-9cb0-1a03db7d812b None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:16.907064 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40dda274-5e24-4fb7-9cb0-1a03db7d812b None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:16.932492 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 476/933] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:32:16 2022] POST /identity/v3/endpoints => generated 347 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:32:16.979831 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cac4a78-9961-4815-aec6-bf3054d14676 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:17.029854 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d844d8-dc73-4fb5-894a-52a448ef2858 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:17.030551 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d844d8-dc73-4fb5-894a-52a448ef2858 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:17.030838 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d844d8-dc73-4fb5-894a-52a448ef2858 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:17.031114 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d844d8-dc73-4fb5-894a-52a448ef2858 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:17.045145 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3d844d8-dc73-4fb5-894a-52a448ef2858 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:32:17.050540 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-b3d844d8-dc73-4fb5-894a-52a448ef2858 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 28 09:32:17.050540 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:17.050540 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:17.050540 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:17.050540 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:17.050540 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:17.050540 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:17.050540 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:17.050540 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:17.050540 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:17.050540 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:17.050540 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:17.050540 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:32:17.050540 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:32:17.050540 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:32:17.050540 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:32:17.051559 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:32:17.051559 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:32:17.051559 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:32:17.051559 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:32:17.051559 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:17.051559 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:17.051559 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:17.051559 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:17.051559 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:17.051559 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:17.051559 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:17.051559 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:17.051559 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:17.051559 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:17.051559 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:17.051559 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:17.052409 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:17.052409 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:17.052409 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:17.052409 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:17.052409 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:32:17.052409 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:32:17.052409 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:32:17.052409 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:32:17.052409 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:32:17.052409 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:32:17.052409 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 28 09:32:17.052409 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:17.052409 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 458/934] 10.222.0.25 () {56 vars in 1120 bytes} [Fri Oct 28 09:32:16 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:32:17.060437 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40dda274-5e24-4fb7-9cb0-1a03db7d812b None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:17.188726 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40dda274-5e24-4fb7-9cb0-1a03db7d812b None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:17.228775 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9eb13ff-2504-44bb-95ae-0c7e32266fd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:17.229401 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9eb13ff-2504-44bb-95ae-0c7e32266fd2 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:17.229720 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9eb13ff-2504-44bb-95ae-0c7e32266fd2 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:17.229966 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9eb13ff-2504-44bb-95ae-0c7e32266fd2 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:17.230725 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9eb13ff-2504-44bb-95ae-0c7e32266fd2 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:17.231038 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9eb13ff-2504-44bb-95ae-0c7e32266fd2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:17.232411 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9eb13ff-2504-44bb-95ae-0c7e32266fd2 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:17.242380 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 477/935] 10.222.0.25 () {56 vars in 1115 bytes} [Fri Oct 28 09:32:17 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:17.248138 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d844d8-dc73-4fb5-894a-52a448ef2858 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:17.328218 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:17.364329 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d844d8-dc73-4fb5-894a-52a448ef2858 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:17.403404 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48c0647-ae4e-40ef-976f-ac25bbb745ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:17.404438 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48c0647-ae4e-40ef-976f-ac25bbb745ef None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:17.405001 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48c0647-ae4e-40ef-976f-ac25bbb745ef None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:17.405588 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f48c0647-ae4e-40ef-976f-ac25bbb745ef None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:17.418809 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f48c0647-ae4e-40ef-976f-ac25bbb745ef None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:17.420652 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f48c0647-ae4e-40ef-976f-ac25bbb745ef None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:17.436908 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-f48c0647-ae4e-40ef-976f-ac25bbb745ef None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:32:17.436908 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:17.436908 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:17.436908 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:17.436908 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:17.436908 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:17.436908 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:17.436908 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:17.436908 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:17.436908 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:17.436908 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:17.436908 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:17.436908 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:32:17.436908 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:32:17.436908 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:32:17.436908 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:32:17.437942 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:17.437942 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:17.437942 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:17.437942 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:17.437942 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:17.437942 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:17.437942 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:17.437942 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:17.437942 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:17.437942 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:17.437942 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:17.437942 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:17.437942 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:17.437942 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:17.437942 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:17.437942 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:17.437942 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:32:17.438915 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:32:17.438915 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:32:17.438915 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:17.441070 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 459/936] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:32:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:32:17.447781 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9eb13ff-2504-44bb-95ae-0c7e32266fd2 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:17.492911 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:17.569638 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9eb13ff-2504-44bb-95ae-0c7e32266fd2 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:17.609256 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed7806d-43f2-4a2a-a2f1-e95b86387a0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:17.609883 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed7806d-43f2-4a2a-a2f1-e95b86387a0d None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:17.610166 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed7806d-43f2-4a2a-a2f1-e95b86387a0d None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:17.610460 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed7806d-43f2-4a2a-a2f1-e95b86387a0d None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:17.611108 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ed7806d-43f2-4a2a-a2f1-e95b86387a0d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:17.611415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ed7806d-43f2-4a2a-a2f1-e95b86387a0d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:17.612774 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ed7806d-43f2-4a2a-a2f1-e95b86387a0d None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:17.625948 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 478/937] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:32:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:17.632241 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48c0647-ae4e-40ef-976f-ac25bbb745ef None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:17.684191 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb4837e-4177-4db1-a01f-4e78a19bc932 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:17.684490 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb4837e-4177-4db1-a01f-4e78a19bc932 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:17.684756 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb4837e-4177-4db1-a01f-4e78a19bc932 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:17.687552 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 479/938] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:32:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 09:32:17.692136 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca340e3-430d-4b0d-9030-caf5b3d299fe None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:17.692345 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca340e3-430d-4b0d-9030-caf5b3d299fe None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:17.692625 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ca340e3-430d-4b0d-9030-caf5b3d299fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:17.711714 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-1ca340e3-430d-4b0d-9030-caf5b3d299fe None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:32:17.713099 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-1ca340e3-430d-4b0d-9030-caf5b3d299fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:32:17.749866 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f48c0647-ae4e-40ef-976f-ac25bbb745ef None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:17.777155 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:17.788794 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e23611-b202-450d-8ceb-7b9f43ce41cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:17.789704 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e23611-b202-450d-8ceb-7b9f43ce41cc None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:17.789987 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e23611-b202-450d-8ceb-7b9f43ce41cc None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:17.790256 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e23611-b202-450d-8ceb-7b9f43ce41cc None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:17.798631 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75e23611-b202-450d-8ceb-7b9f43ce41cc None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:17.800123 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75e23611-b202-450d-8ceb-7b9f43ce41cc None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:17.808327 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 480/939] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:32:17 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 09:32:17.816033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-75e23611-b202-450d-8ceb-7b9f43ce41cc None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 28 09:32:17.816033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:17.816033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:17.816033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:17.816033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:17.816033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:17.816033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:17.816033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:17.816033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:17.816033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:17.816033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:17.816033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:17.816033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 09:32:17.816033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 09:32:17.816033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 09:32:17.816033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 09:32:17.817181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:17.817181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:17.817181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 09:32:17.817181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:32:17.817181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 09:32:17.817181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:32:17.817181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:17.817181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:17.817181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:17.817181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:17.817181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:17.817181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:17.817181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:17.817181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:17.817181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:17.817181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:17.817181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:17.817181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:17.818192 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:17.818192 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:17.818192 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 09:32:17.818192 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 09:32:17.818192 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 09:32:17.818192 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 09:32:17.818192 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 09:32:17.818192 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 09:32:17.818192 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 28 09:32:17.818192 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:17.820395 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 460/940] 10.222.0.25 () {56 vars in 1117 bytes} [Fri Oct 28 09:32:17 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:17.826232 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ca340e3-430d-4b0d-9030-caf5b3d299fe None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:17.858856 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:17.932867 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ca340e3-430d-4b0d-9030-caf5b3d299fe None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:17.970552 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290277a1-61fd-472c-8006-ebd97f79502e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:17.971180 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290277a1-61fd-472c-8006-ebd97f79502e None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:17.971505 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290277a1-61fd-472c-8006-ebd97f79502e None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:17.971787 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290277a1-61fd-472c-8006-ebd97f79502e None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:17.972612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290277a1-61fd-472c-8006-ebd97f79502e None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:17.972924 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290277a1-61fd-472c-8006-ebd97f79502e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:17.974498 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290277a1-61fd-472c-8006-ebd97f79502e None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:17.994850 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 481/941] 10.222.0.25 () {56 vars in 1149 bytes} [Fri Oct 28 09:32:17 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:18.000511 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e23611-b202-450d-8ceb-7b9f43ce41cc None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:18.109299 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75e23611-b202-450d-8ceb-7b9f43ce41cc None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:18.109694 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290277a1-61fd-472c-8006-ebd97f79502e None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:18.154007 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801f7c97-2fc2-47d3-b198-eaedfe128dec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:18.154673 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801f7c97-2fc2-47d3-b198-eaedfe128dec None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:18.154958 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801f7c97-2fc2-47d3-b198-eaedfe128dec None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:18.155262 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801f7c97-2fc2-47d3-b198-eaedfe128dec None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:18.170035 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-801f7c97-2fc2-47d3-b198-eaedfe128dec None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:18.171573 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-801f7c97-2fc2-47d3-b198-eaedfe128dec None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:18.190823 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-801f7c97-2fc2-47d3-b198-eaedfe128dec None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:32:18.190823 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:18.190823 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:18.190823 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:18.190823 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:18.190823 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:18.190823 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:18.190823 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:18.190823 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:18.190823 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:18.190823 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:18.190823 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:18.190823 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:32:18.190823 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:32:18.190823 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:32:18.190823 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:32:18.191900 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:18.191900 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:18.191900 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:18.191900 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:18.191900 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:18.191900 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:18.191900 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:18.191900 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:18.191900 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:18.191900 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:18.191900 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:18.191900 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:18.191900 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:18.191900 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:18.191900 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:18.191900 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:18.191900 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:32:18.193437 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:32:18.193437 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:32:18.193437 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:18.193437 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 461/942] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:32:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:18.199248 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801f7c97-2fc2-47d3-b198-eaedfe128dec None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:18.214685 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:18.236153 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290277a1-61fd-472c-8006-ebd97f79502e None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:18.246922 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:18.285270 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4077f67-6f9a-4c8e-b2de-1cd4a2e39df8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:18.294755 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4077f67-6f9a-4c8e-b2de-1cd4a2e39df8 None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:18.294755 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4077f67-6f9a-4c8e-b2de-1cd4a2e39df8 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:18.294755 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4077f67-6f9a-4c8e-b2de-1cd4a2e39df8 None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:18.294755 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4077f67-6f9a-4c8e-b2de-1cd4a2e39df8 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:18.294755 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4077f67-6f9a-4c8e-b2de-1cd4a2e39df8 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:18.298179 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: WARNING keystone.server.flask.common [None req-d4077f67-6f9a-4c8e-b2de-1cd4a2e39df8 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Oct 28 09:32:18.325411 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.sql.core [None req-d4077f67-6f9a-4c8e-b2de-1cd4a2e39df8 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 28 09:32:18.325411 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 28 09:32:18.325411 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [parameters: {'id': 'ca6be1b8905c4ea4b37710996744b56e', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Oct 28 09:32:18.325411 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81577) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 28 09:32:18.343954 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801f7c97-2fc2-47d3-b198-eaedfe128dec None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:18.348371 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-d4077f67-6f9a-4c8e-b2de-1cd4a2e39df8 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id ca6be1b8905c4ea4b37710996744b56e.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id ca6be1b8905c4ea4b37710996744b56e. Oct 28 09:32:18.348371 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:18.348371 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 28 09:32:18.348371 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 28 09:32:18.348371 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 28 09:32:18.348371 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 28 09:32:18.348371 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 28 09:32:18.348371 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application result = self._query(query) Oct 28 09:32:18.348371 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 28 09:32:18.348371 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application conn.query(q) Oct 28 09:32:18.348371 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 28 09:32:18.348371 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 28 09:32:18.348371 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 28 09:32:18.349265 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application result.read() Oct 28 09:32:18.349265 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 28 09:32:18.349265 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 28 09:32:18.349265 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 28 09:32:18.349265 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 28 09:32:18.349265 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 28 09:32:18.349265 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 28 09:32:18.349265 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 28 09:32:18.349265 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 28 09:32:18.349265 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 28 09:32:18.349265 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:18.349265 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 28 09:32:18.349265 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:18.349265 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:18.349265 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 28 09:32:18.349265 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 28 09:32:18.349265 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Oct 28 09:32:18.349265 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return project_ref.to_dict() Oct 28 09:32:18.357263 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 28 09:32:18.357263 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application next(self.gen) Oct 28 09:32:18.357263 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 28 09:32:18.357263 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application yield resource Oct 28 09:32:18.357263 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 28 09:32:18.357263 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application next(self.gen) Oct 28 09:32:18.357263 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 28 09:32:18.357263 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self.session.rollback() Oct 28 09:32:18.357263 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 28 09:32:18.357263 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self.force_reraise() Oct 28 09:32:18.357263 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 28 09:32:18.357263 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise self.value Oct 28 09:32:18.357263 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 28 09:32:18.357263 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 28 09:32:18.357263 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 28 09:32:18.357263 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application session.commit() Oct 28 09:32:18.357263 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 28 09:32:18.359990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 28 09:32:18.359990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 28 09:32:18.359990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._prepare_impl() Oct 28 09:32:18.359990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 28 09:32:18.359990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self.session.flush() Oct 28 09:32:18.359990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 28 09:32:18.359990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._flush(objects) Oct 28 09:32:18.359990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 28 09:32:18.359990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 28 09:32:18.359990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 28 09:32:18.359990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application compat.raise_( Oct 28 09:32:18.359990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 28 09:32:18.359990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception Oct 28 09:32:18.359990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 28 09:32:18.359990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application flush_context.execute() Oct 28 09:32:18.359990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 28 09:32:18.359990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rec.execute(self) Oct 28 09:32:18.362044 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 28 09:32:18.362044 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 28 09:32:18.362044 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 28 09:32:18.362044 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 28 09:32:18.362044 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Oct 28 09:32:18.362044 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application c = connection._execute_20( Oct 28 09:32:18.362044 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 28 09:32:18.362044 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 28 09:32:18.362044 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 28 09:32:18.362044 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 28 09:32:18.362044 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 28 09:32:18.362044 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 28 09:32:18.362044 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 28 09:32:18.362044 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 28 09:32:18.362044 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 28 09:32:18.364023 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 28 09:32:18.364023 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 28 09:32:18.364023 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception Oct 28 09:32:18.364023 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 28 09:32:18.364023 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 28 09:32:18.364023 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 28 09:32:18.364023 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 28 09:32:18.364023 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 28 09:32:18.364023 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application result = self._query(query) Oct 28 09:32:18.364023 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 28 09:32:18.364023 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application conn.query(q) Oct 28 09:32:18.364023 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 28 09:32:18.364023 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 28 09:32:18.364023 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 28 09:32:18.364023 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application result.read() Oct 28 09:32:18.364023 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 28 09:32:18.364023 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 28 09:32:18.367698 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 28 09:32:18.367698 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 28 09:32:18.367698 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 28 09:32:18.367698 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 28 09:32:18.367698 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 28 09:32:18.367698 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 28 09:32:18.367698 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Oct 28 09:32:18.367698 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Oct 28 09:32:18.367698 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [parameters: {'id': 'ca6be1b8905c4ea4b37710996744b56e', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Oct 28 09:32:18.367698 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 28 09:32:18.367698 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:18.367698 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 09:32:18.367698 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:18.367698 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:18.367698 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Oct 28 09:32:18.369614 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Oct 28 09:32:18.369614 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 28 09:32:18.369614 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 28 09:32:18.369614 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Oct 28 09:32:18.369614 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:18.369614 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 09:32:18.369614 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:18.369614 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:18.369614 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:18.369614 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:18.369614 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:18.369614 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:18.369614 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:18.369614 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:18.369614 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:18.369614 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:18.369614 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:18.371759 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:18.371759 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Oct 28 09:32:18.371759 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Oct 28 09:32:18.371759 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:18.371759 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:18.371759 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Oct 28 09:32:18.371759 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.Conflict( Oct 28 09:32:18.371759 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id ca6be1b8905c4ea4b37710996744b56e. Oct 28 09:32:18.371759 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:18.371759 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 482/943] 10.222.0.25 () {60 vars in 1140 bytes} [Fri Oct 28 09:32:18 2022] POST /identity/v3/projects => generated 262 bytes in 241 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:32:18.371759 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4077f67-6f9a-4c8e-b2de-1cd4a2e39df8 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:18.401694 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e6df80-25fa-4e70-a08a-fbf423d192dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:18.402319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e6df80-25fa-4e70-a08a-fbf423d192dc None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:18.402634 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e6df80-25fa-4e70-a08a-fbf423d192dc None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:18.402916 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e6df80-25fa-4e70-a08a-fbf423d192dc None admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:18.403667 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e6df80-25fa-4e70-a08a-fbf423d192dc None admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:18.403999 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e6df80-25fa-4e70-a08a-fbf423d192dc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:18.405384 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e6df80-25fa-4e70-a08a-fbf423d192dc None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:18.425691 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 462/944] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:32:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:18.426827 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e6df80-25fa-4e70-a08a-fbf423d192dc None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:18.528323 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:18.536604 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e6df80-25fa-4e70-a08a-fbf423d192dc None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:18.537678 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4077f67-6f9a-4c8e-b2de-1cd4a2e39df8 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:18.580552 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe2d22b-8571-4bee-a7f8-7af6bb316a2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:18.581232 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe2d22b-8571-4bee-a7f8-7af6bb316a2d None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:18.581780 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe2d22b-8571-4bee-a7f8-7af6bb316a2d None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:18.582057 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe2d22b-8571-4bee-a7f8-7af6bb316a2d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:18.591883 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbe2d22b-8571-4bee-a7f8-7af6bb316a2d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:18.593732 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbe2d22b-8571-4bee-a7f8-7af6bb316a2d None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:18.594143 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f38f6d2-1974-4597-b449-4b78a5806f2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:18.595172 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f38f6d2-1974-4597-b449-4b78a5806f2d None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:18.595675 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f38f6d2-1974-4597-b449-4b78a5806f2d None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:18.596288 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f38f6d2-1974-4597-b449-4b78a5806f2d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:18.604531 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f38f6d2-1974-4597-b449-4b78a5806f2d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:18.606560 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f38f6d2-1974-4597-b449-4b78a5806f2d None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:18.609466 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-cbe2d22b-8571-4bee-a7f8-7af6bb316a2d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:32:18.609466 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:18.609466 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:18.609466 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:18.609466 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:18.609466 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:18.609466 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:18.609466 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:18.609466 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:18.609466 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:18.609466 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:18.609466 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:18.609466 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 09:32:18.609466 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 09:32:18.609466 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 09:32:18.610364 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 09:32:18.610364 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:18.610364 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:18.610364 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:18.610364 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:18.610364 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:18.610364 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:18.610364 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:18.610364 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:18.610364 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:18.610364 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:18.610364 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:18.610364 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:18.610364 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:18.610364 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:18.610364 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:18.610364 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:18.611340 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 09:32:18.611340 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 09:32:18.611340 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 09:32:18.611340 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 09:32:18.611340 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 09:32:18.611340 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 09:32:18.611340 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:32:18.611340 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:18.617523 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 463/945] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:32:18 2022] GET /identity/v3/projects/service => generated 88 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:32:18.618705 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe2d22b-8571-4bee-a7f8-7af6bb316a2d None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:18.624415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-0f38f6d2-1974-4597-b449-4b78a5806f2d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:32:18.624415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:18.624415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:18.624415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:18.624415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:18.624415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:18.624415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:18.624415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:18.624415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:18.624415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:18.624415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:18.624415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:18.624415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 09:32:18.624415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 09:32:18.624415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 09:32:18.625340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 09:32:18.625340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:18.625340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:18.625340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:18.625340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:18.625340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:18.625340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:18.625340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:18.625340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:18.625340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:18.625340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:18.625340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:18.625340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:18.625340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:18.625340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:18.625340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:18.625340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:18.626216 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 09:32:18.626216 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 09:32:18.626216 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 09:32:18.626216 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 09:32:18.626216 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 09:32:18.626216 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 09:32:18.626216 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:32:18.626216 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:18.629554 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 483/946] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:32:18 2022] GET /identity/v3/projects/service => generated 88 bytes in 270 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:18.636205 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f38f6d2-1974-4597-b449-4b78a5806f2d None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:18.640052 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:18.731662 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe2d22b-8571-4bee-a7f8-7af6bb316a2d None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:18.764647 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f38f6d2-1974-4597-b449-4b78a5806f2d None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:18.770015 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6365d446-41ed-4eb9-8f2b-a884bcc1e17d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:18.770692 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6365d446-41ed-4eb9-8f2b-a884bcc1e17d None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:18.770976 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6365d446-41ed-4eb9-8f2b-a884bcc1e17d None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:18.771255 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6365d446-41ed-4eb9-8f2b-a884bcc1e17d None admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:18.772525 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6365d446-41ed-4eb9-8f2b-a884bcc1e17d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:18.772841 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6365d446-41ed-4eb9-8f2b-a884bcc1e17d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:18.774342 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6365d446-41ed-4eb9-8f2b-a884bcc1e17d None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:18.793338 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 464/947] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Oct 28 09:32:18 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:18.794212 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6365d446-41ed-4eb9-8f2b-a884bcc1e17d None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:18.809755 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4d4d59-8c30-45ec-a233-e59c9522164b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:18.811187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4d4d59-8c30-45ec-a233-e59c9522164b None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:18.811722 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4d4d59-8c30-45ec-a233-e59c9522164b None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:18.812215 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4d4d59-8c30-45ec-a233-e59c9522164b None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:18.813156 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:18.813670 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c4d4d59-8c30-45ec-a233-e59c9522164b None admin] RBAC: Adding query filter params (name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:18.814861 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c4d4d59-8c30-45ec-a233-e59c9522164b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:18.816733 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c4d4d59-8c30-45ec-a233-e59c9522164b None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:18.843388 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 484/948] 10.222.0.25 () {56 vars in 1112 bytes} [Fri Oct 28 09:32:18 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:32:18.879600 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda9dd26-6dfc-4845-9609-4a2ad5a5018a None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:18.879909 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda9dd26-6dfc-4845-9609-4a2ad5a5018a None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:18.880292 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda9dd26-6dfc-4845-9609-4a2ad5a5018a None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:18.882755 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 485/949] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:32:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 09:32:18.891481 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24880a3c-b816-4986-992a-9ab1acee2fbd None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:18.892023 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24880a3c-b816-4986-992a-9ab1acee2fbd None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:18.892538 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24880a3c-b816-4986-992a-9ab1acee2fbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:18.913661 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-24880a3c-b816-4986-992a-9ab1acee2fbd None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:32:18.915725 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-24880a3c-b816-4986-992a-9ab1acee2fbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:32:18.918890 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6365d446-41ed-4eb9-8f2b-a884bcc1e17d None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:18.970302 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235aef74-2ee3-4916-a3a4-c3da1ea034d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:18.971420 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235aef74-2ee3-4916-a3a4-c3da1ea034d8 None admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:18.971918 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235aef74-2ee3-4916-a3a4-c3da1ea034d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:18.972415 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235aef74-2ee3-4916-a3a4-c3da1ea034d8 None admin] PATH_INFO: `/v3/projects/455d1c826b974bc38497d18a516867c7/users/97c8877c74194a6bbbf25b3c8f5b7dbd/roles/4e19a68eecb04cb998e6aa7adfb48ad3` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:18.975506 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-235aef74-2ee3-4916-a3a4-c3da1ea034d8 None admin] RBAC: Authorizing `identity:create_grant(project_id=455d1c826b974bc38497d18a516867c7, user_id=97c8877c74194a6bbbf25b3c8f5b7dbd, role_id=4e19a68eecb04cb998e6aa7adfb48ad3)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:18.977233 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-235aef74-2ee3-4916-a3a4-c3da1ea034d8 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:18.996146 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:18.998839 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 465/950] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Oct 28 09:32:18 2022] PUT /identity/v3/projects/455d1c826b974bc38497d18a516867c7/users/97c8877c74194a6bbbf25b3c8f5b7dbd/roles/4e19a68eecb04cb998e6aa7adfb48ad3 => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:32:19.010603 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 486/951] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:32:18 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 09:32:19.292055 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235aef74-2ee3-4916-a3a4-c3da1ea034d8 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:19.300440 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f50424e-efb3-4b15-b019-c953dc054cdb None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:19.300802 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f50424e-efb3-4b15-b019-c953dc054cdb None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:19.301010 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f50424e-efb3-4b15-b019-c953dc054cdb None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:19.302756 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 487/952] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:32:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 09:32:19.308960 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4ab95b-ad49-42b3-a1e0-73f8ab862d57 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:19.309288 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4ab95b-ad49-42b3-a1e0-73f8ab862d57 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:19.309528 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4ab95b-ad49-42b3-a1e0-73f8ab862d57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:19.329350 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-7a4ab95b-ad49-42b3-a1e0-73f8ab862d57 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:32:19.330619 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7a4ab95b-ad49-42b3-a1e0-73f8ab862d57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:32:19.385422 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:19.406519 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 488/953] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:32:19 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 09:32:19.416036 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235aef74-2ee3-4916-a3a4-c3da1ea034d8 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:19.453428 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0365867-ef36-49a5-857c-7969177967b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:19.454525 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0365867-ef36-49a5-857c-7969177967b5 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:19.454967 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0365867-ef36-49a5-857c-7969177967b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:19.455246 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0365867-ef36-49a5-857c-7969177967b5 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:19.455841 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0365867-ef36-49a5-857c-7969177967b5 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:19.457226 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0365867-ef36-49a5-857c-7969177967b5 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:19.467296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-b0365867-ef36-49a5-857c-7969177967b5 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 09:32:19.467296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:19.467296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 09:32:19.467296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 09:32:19.467296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 09:32:19.467296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 09:32:19.467296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 09:32:19.467296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:32:19.467296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 09:32:19.467296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:19.467296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 09:32:19.467296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:19.467296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:19.467296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:19.467296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:19.467296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:19.467296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:19.468485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:19.468485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:19.468485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:19.468485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:19.468485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:19.468485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:19.468485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 09:32:19.468485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 09:32:19.468485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 09:32:19.468485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 09:32:19.468485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:19.468485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:19.468485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:19.468485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:19.468485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:19.468485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:19.469427 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:19.469427 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:19.469427 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:19.469427 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:19.469427 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:19.469427 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:19.469427 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:19.469427 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:19.469427 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:19.469427 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:19.469427 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 09:32:19.469427 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:32:19.469427 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 09:32:19.469427 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:19.473400 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 466/954] 10.222.0.25 () {56 vars in 1126 bytes} [Fri Oct 28 09:32:19 2022] GET /identity/v3/services/object-store => generated 93 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:19.474802 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4ab95b-ad49-42b3-a1e0-73f8ab862d57 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:19.592738 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4ab95b-ad49-42b3-a1e0-73f8ab862d57 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:19.616347 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:19.628343 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c4f645-9f67-45ae-b633-71451bef54d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:19.628952 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c4f645-9f67-45ae-b633-71451bef54d2 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:19.629230 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c4f645-9f67-45ae-b633-71451bef54d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:19.629504 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c4f645-9f67-45ae-b633-71451bef54d2 None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:19.630133 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1c4f645-9f67-45ae-b633-71451bef54d2 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:19.630460 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1c4f645-9f67-45ae-b633-71451bef54d2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:19.632512 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1c4f645-9f67-45ae-b633-71451bef54d2 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:19.642514 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 489/955] 10.222.0.25 () {56 vars in 1122 bytes} [Fri Oct 28 09:32:19 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:32:19.644772 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0365867-ef36-49a5-857c-7969177967b5 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:19.677148 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c4f645-9f67-45ae-b633-71451bef54d2 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:19.766303 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0365867-ef36-49a5-857c-7969177967b5 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:19.805603 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e607900c-c8cd-4075-91de-18a866fc77f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:19.806559 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e607900c-c8cd-4075-91de-18a866fc77f1 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:19.807124 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e607900c-c8cd-4075-91de-18a866fc77f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:19.807606 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e607900c-c8cd-4075-91de-18a866fc77f1 None admin] PATH_INFO: `/v3/services` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:19.808469 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e607900c-c8cd-4075-91de-18a866fc77f1 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:19.809014 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e607900c-c8cd-4075-91de-18a866fc77f1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:19.810652 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e607900c-c8cd-4075-91de-18a866fc77f1 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:19.823346 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 467/956] 10.222.0.25 () {56 vars in 1122 bytes} [Fri Oct 28 09:32:19 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:19.833995 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e607900c-c8cd-4075-91de-18a866fc77f1 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:19.844178 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:19.853207 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c4f645-9f67-45ae-b633-71451bef54d2 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:19.927110 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8425198-09ac-4583-854e-18ef3b22dfd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:19.927752 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8425198-09ac-4583-854e-18ef3b22dfd9 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:19.928073 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8425198-09ac-4583-854e-18ef3b22dfd9 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:19.928349 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8425198-09ac-4583-854e-18ef3b22dfd9 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:19.928938 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8425198-09ac-4583-854e-18ef3b22dfd9 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:19.930307 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8425198-09ac-4583-854e-18ef3b22dfd9 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:19.944814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-a8425198-09ac-4583-854e-18ef3b22dfd9 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 28 09:32:19.944814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:19.944814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 09:32:19.944814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 09:32:19.944814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 09:32:19.944814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 09:32:19.944814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 09:32:19.944814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:32:19.944814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 28 09:32:19.944814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:19.944814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 09:32:19.944814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:19.944814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:19.944814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:19.944814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:19.944814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:19.944814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:19.945911 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:19.945911 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:19.945911 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:19.945911 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:19.945911 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:19.945911 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:19.945911 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 09:32:19.945911 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 09:32:19.945911 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 09:32:19.945911 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 09:32:19.945911 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:19.945911 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:19.945911 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:19.945911 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:19.945911 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:19.945911 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:19.946936 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:19.946936 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:19.946936 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:19.946936 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:19.946936 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:19.946936 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:19.946936 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:19.946936 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:19.946936 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:19.946936 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:19.946936 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 09:32:19.946936 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:32:19.946936 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 28 09:32:19.946936 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:19.951494 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 490/957] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:32:19 2022] GET /identity/v3/services/compute => generated 88 bytes in 275 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:32:19.952772 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8425198-09ac-4583-854e-18ef3b22dfd9 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:20.012774 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e607900c-c8cd-4075-91de-18a866fc77f1 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:20.019126 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:20.064905 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92cea7bf-c3db-4cc6-945a-f65857b44c96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:20.066002 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92cea7bf-c3db-4cc6-945a-f65857b44c96 None admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:20.066545 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92cea7bf-c3db-4cc6-945a-f65857b44c96 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:20.067136 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92cea7bf-c3db-4cc6-945a-f65857b44c96 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:20.068172 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92cea7bf-c3db-4cc6-945a-f65857b44c96 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:20.070879 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92cea7bf-c3db-4cc6-945a-f65857b44c96 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:20.101013 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8425198-09ac-4583-854e-18ef3b22dfd9 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:20.107223 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 468/958] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:32:19 2022] POST /identity/v3/endpoints => generated 349 bytes in 275 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:32:20.164086 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca0c5ede-4ad1-4ee9-9ff2-4a73df12f25f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:20.164725 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0c5ede-4ad1-4ee9-9ff2-4a73df12f25f None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:20.165005 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0c5ede-4ad1-4ee9-9ff2-4a73df12f25f None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:20.165277 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0c5ede-4ad1-4ee9-9ff2-4a73df12f25f None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:20.165919 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca0c5ede-4ad1-4ee9-9ff2-4a73df12f25f None admin] RBAC: Adding query filter params (name=compute) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:20.166228 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca0c5ede-4ad1-4ee9-9ff2-4a73df12f25f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:20.167745 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca0c5ede-4ad1-4ee9-9ff2-4a73df12f25f None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:20.174115 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 491/959] 10.222.0.25 () {56 vars in 1112 bytes} [Fri Oct 28 09:32:19 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:20.180688 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92cea7bf-c3db-4cc6-945a-f65857b44c96 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:20.252557 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:20.312535 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92cea7bf-c3db-4cc6-945a-f65857b44c96 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:20.370265 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-509cc52e-5dd3-4ae1-8e9f-8f68a51c70b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:20.371132 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509cc52e-5dd3-4ae1-8e9f-8f68a51c70b7 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:20.371547 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509cc52e-5dd3-4ae1-8e9f-8f68a51c70b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:20.371962 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509cc52e-5dd3-4ae1-8e9f-8f68a51c70b7 None admin] PATH_INFO: `/v3/services` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:20.372936 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-509cc52e-5dd3-4ae1-8e9f-8f68a51c70b7 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:20.373383 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-509cc52e-5dd3-4ae1-8e9f-8f68a51c70b7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:20.375597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-509cc52e-5dd3-4ae1-8e9f-8f68a51c70b7 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:20.383899 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 469/960] 10.222.0.25 () {56 vars in 1112 bytes} [Fri Oct 28 09:32:20 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:21.265299 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963839f4-0c9f-459a-882b-1ea834cd92a0 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:21.265845 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963839f4-0c9f-459a-882b-1ea834cd92a0 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:21.266312 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963839f4-0c9f-459a-882b-1ea834cd92a0 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:21.268782 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 492/961] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:32:21 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:32:21.275216 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e96db6-eb0b-411d-bf5c-3ca009cc934b None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:21.275510 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e96db6-eb0b-411d-bf5c-3ca009cc934b None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:21.275791 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e96db6-eb0b-411d-bf5c-3ca009cc934b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:21.311510 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-83e96db6-eb0b-411d-bf5c-3ca009cc934b None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:32:21.312919 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-83e96db6-eb0b-411d-bf5c-3ca009cc934b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:32:21.379507 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ae5868-3e25-40e3-b187-d16792a13e1e None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:21.379844 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ae5868-3e25-40e3-b187-d16792a13e1e None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:21.380047 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ae5868-3e25-40e3-b187-d16792a13e1e None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:21.382302 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 493/962] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:32:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:32:21.388603 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b4037a-674a-4b20-8892-91125249a2e5 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:21.388913 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b4037a-674a-4b20-8892-91125249a2e5 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:21.389143 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b4037a-674a-4b20-8892-91125249a2e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:21.410693 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-19b4037a-674a-4b20-8892-91125249a2e5 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:32:21.412139 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-19b4037a-674a-4b20-8892-91125249a2e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:32:21.423708 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 470/963] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:32:21 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:32:21.566531 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 494/964] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:32:21 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:32:21.837240 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e96db6-eb0b-411d-bf5c-3ca009cc934b None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:21.907690 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:21.952785 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e96db6-eb0b-411d-bf5c-3ca009cc934b None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:21.978192 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b4037a-674a-4b20-8892-91125249a2e5 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:21.992672 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126279cf-b6f4-4ecd-a140-c651cf4c3da7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:21.994027 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126279cf-b6f4-4ecd-a140-c651cf4c3da7 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:21.994316 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126279cf-b6f4-4ecd-a140-c651cf4c3da7 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:21.998427 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126279cf-b6f4-4ecd-a140-c651cf4c3da7 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:22.010001 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-126279cf-b6f4-4ecd-a140-c651cf4c3da7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:32:22.015717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-126279cf-b6f4-4ecd-a140-c651cf4c3da7 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 28 09:32:22.015717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:22.015717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:22.015717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:22.015717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:22.015717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:22.015717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:22.015717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:22.015717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:22.015717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:22.015717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:22.015717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:22.015717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:32:22.015717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:32:22.015717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:32:22.015717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:32:22.016747 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:32:22.016747 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:32:22.016747 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:32:22.016747 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:32:22.016747 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:22.016747 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:22.016747 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:22.016747 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:22.016747 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:22.016747 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:22.016747 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:22.016747 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:22.016747 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:22.016747 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:22.016747 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:22.016747 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:22.017616 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:22.017616 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:22.017616 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:22.017616 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:22.017616 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:32:22.017616 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:32:22.017616 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:32:22.017616 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:32:22.017616 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:32:22.017616 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:32:22.017616 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 28 09:32:22.017616 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:22.019266 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 471/965] 10.222.0.25 () {56 vars in 1120 bytes} [Fri Oct 28 09:32:21 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:22.024629 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126279cf-b6f4-4ecd-a140-c651cf4c3da7 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:22.062197 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:22.142463 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126279cf-b6f4-4ecd-a140-c651cf4c3da7 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:22.159488 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:22.182763 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b4037a-674a-4b20-8892-91125249a2e5 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:22.183702 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8372856-d2ad-4b1c-84d6-18fbbff68ec5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:22.184337 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8372856-d2ad-4b1c-84d6-18fbbff68ec5 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:22.184621 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8372856-d2ad-4b1c-84d6-18fbbff68ec5 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:22.184900 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8372856-d2ad-4b1c-84d6-18fbbff68ec5 None admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:22.185592 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8372856-d2ad-4b1c-84d6-18fbbff68ec5 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:22.185901 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8372856-d2ad-4b1c-84d6-18fbbff68ec5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:22.187555 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8372856-d2ad-4b1c-84d6-18fbbff68ec5 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:22.204753 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 472/966] 10.222.0.25 () {56 vars in 1115 bytes} [Fri Oct 28 09:32:22 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:22.206433 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8372856-d2ad-4b1c-84d6-18fbbff68ec5 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:22.243343 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8c4277-7da1-464a-be3d-ffc885afa60d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:22.243980 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8c4277-7da1-464a-be3d-ffc885afa60d None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:22.244266 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8c4277-7da1-464a-be3d-ffc885afa60d None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:22.244550 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8c4277-7da1-464a-be3d-ffc885afa60d None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:22.245806 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c8c4277-7da1-464a-be3d-ffc885afa60d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:22.247370 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c8c4277-7da1-464a-be3d-ffc885afa60d None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:22.266731 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.sql.core [None req-0c8c4277-7da1-464a-be3d-ffc885afa60d None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 28 09:32:22.266731 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Oct 28 09:32:22.266731 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [parameters: {'user_id': '999a459065d648d49a379e9f97ca3655', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Oct 28 09:32:22.266731 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81577) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Oct 28 09:32:22.275741 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-0c8c4277-7da1-464a-be3d-ffc885afa60d None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Oct 28 09:32:22.275741 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:22.275741 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 28 09:32:22.275741 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 28 09:32:22.275741 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 28 09:32:22.275741 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 28 09:32:22.275741 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 28 09:32:22.275741 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application result = self._query(query) Oct 28 09:32:22.275741 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 28 09:32:22.275741 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application conn.query(q) Oct 28 09:32:22.275741 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 28 09:32:22.275741 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 28 09:32:22.275741 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 28 09:32:22.275741 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application result.read() Oct 28 09:32:22.275741 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 28 09:32:22.277346 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 28 09:32:22.277346 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 28 09:32:22.277346 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 28 09:32:22.277346 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 28 09:32:22.277346 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 28 09:32:22.277346 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 28 09:32:22.277346 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 28 09:32:22.277346 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 28 09:32:22.277346 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:22.277346 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Oct 28 09:32:22.277346 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:22.277346 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:22.277346 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Oct 28 09:32:22.277346 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return method(*args, **kwargs) Oct 28 09:32:22.277346 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Oct 28 09:32:22.277346 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Oct 28 09:32:22.277346 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 28 09:32:22.277346 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application next(self.gen) Oct 28 09:32:22.279154 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Oct 28 09:32:22.279154 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application yield resource Oct 28 09:32:22.279154 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Oct 28 09:32:22.279154 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application next(self.gen) Oct 28 09:32:22.279154 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Oct 28 09:32:22.279154 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self.session.rollback() Oct 28 09:32:22.279154 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Oct 28 09:32:22.279154 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self.force_reraise() Oct 28 09:32:22.279154 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Oct 28 09:32:22.279154 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise self.value Oct 28 09:32:22.279154 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Oct 28 09:32:22.279154 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Oct 28 09:32:22.279154 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Oct 28 09:32:22.279154 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application session.commit() Oct 28 09:32:22.279154 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Oct 28 09:32:22.279154 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Oct 28 09:32:22.279154 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Oct 28 09:32:22.281891 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._prepare_impl() Oct 28 09:32:22.281891 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Oct 28 09:32:22.281891 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self.session.flush() Oct 28 09:32:22.281891 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Oct 28 09:32:22.281891 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._flush(objects) Oct 28 09:32:22.281891 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Oct 28 09:32:22.281891 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Oct 28 09:32:22.281891 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Oct 28 09:32:22.281891 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application compat.raise_( Oct 28 09:32:22.281891 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 28 09:32:22.281891 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception Oct 28 09:32:22.281891 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Oct 28 09:32:22.281891 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application flush_context.execute() Oct 28 09:32:22.281891 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Oct 28 09:32:22.281891 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rec.execute(self) Oct 28 09:32:22.281891 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Oct 28 09:32:22.281891 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Oct 28 09:32:22.285597 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Oct 28 09:32:22.285597 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application _emit_insert_statements( Oct 28 09:32:22.285597 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Oct 28 09:32:22.285597 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application result = connection._execute_20( Oct 28 09:32:22.285597 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Oct 28 09:32:22.285597 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Oct 28 09:32:22.285597 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Oct 28 09:32:22.285597 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Oct 28 09:32:22.285597 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Oct 28 09:32:22.285597 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ret = self._execute_context( Oct 28 09:32:22.285597 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Oct 28 09:32:22.285597 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Oct 28 09:32:22.285597 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Oct 28 09:32:22.285597 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Oct 28 09:32:22.285597 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Oct 28 09:32:22.285597 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception Oct 28 09:32:22.291136 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Oct 28 09:32:22.291136 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self.dialect.do_execute( Oct 28 09:32:22.291136 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Oct 28 09:32:22.291136 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Oct 28 09:32:22.291136 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Oct 28 09:32:22.291136 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application result = self._query(query) Oct 28 09:32:22.291136 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Oct 28 09:32:22.291136 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application conn.query(q) Oct 28 09:32:22.291136 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Oct 28 09:32:22.291136 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Oct 28 09:32:22.291136 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Oct 28 09:32:22.291136 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application result.read() Oct 28 09:32:22.291136 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Oct 28 09:32:22.291136 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Oct 28 09:32:22.291136 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Oct 28 09:32:22.291136 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application packet.raise_for_error() Oct 28 09:32:22.296409 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Oct 28 09:32:22.296409 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Oct 28 09:32:22.296409 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Oct 28 09:32:22.296409 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Oct 28 09:32:22.296409 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Oct 28 09:32:22.296409 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Oct 28 09:32:22.296409 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [parameters: {'user_id': '999a459065d648d49a379e9f97ca3655', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Oct 28 09:32:22.296409 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Oct 28 09:32:22.296409 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:22.296409 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 09:32:22.296409 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:22.296409 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:22.296409 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:22.296409 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:22.296409 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:22.300142 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:22.300142 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:22.300142 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:22.300142 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:22.300142 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:22.300142 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:22.300142 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:22.300142 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Oct 28 09:32:22.300142 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Oct 28 09:32:22.300142 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:22.300142 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:22.300142 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 09:32:22.300142 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:32:22.300142 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 09:32:22.300142 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:32:22.300142 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Oct 28 09:32:22.300142 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Oct 28 09:32:22.305377 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Oct 28 09:32:22.305377 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Oct 28 09:32:22.305377 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Oct 28 09:32:22.305377 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Oct 28 09:32:22.305377 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Oct 28 09:32:22.305377 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:22.305377 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 495/967] 10.222.0.25 () {60 vars in 1131 bytes} [Fri Oct 28 09:32:21 2022] POST /identity/v3/users => generated 160 bytes in 300 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:22.305377 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8c4277-7da1-464a-be3d-ffc885afa60d None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:22.440364 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8c4277-7da1-464a-be3d-ffc885afa60d None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:22.442678 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:22.463488 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8372856-d2ad-4b1c-84d6-18fbbff68ec5 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:22.505040 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9308395-2c5e-4505-9368-dd7f3cadaf6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:22.505708 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9308395-2c5e-4505-9368-dd7f3cadaf6d None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:22.505990 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9308395-2c5e-4505-9368-dd7f3cadaf6d None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:22.506343 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9308395-2c5e-4505-9368-dd7f3cadaf6d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:22.520243 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9308395-2c5e-4505-9368-dd7f3cadaf6d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:22.522483 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9308395-2c5e-4505-9368-dd7f3cadaf6d None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:22.528224 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a81bf3-6248-42cf-982f-85c9d9170b94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:22.529902 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a81bf3-6248-42cf-982f-85c9d9170b94 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:22.530799 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a81bf3-6248-42cf-982f-85c9d9170b94 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:22.531560 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a81bf3-6248-42cf-982f-85c9d9170b94 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:22.539361 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-e9308395-2c5e-4505-9368-dd7f3cadaf6d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:32:22.539361 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:22.539361 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:22.539361 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:22.539361 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:22.539361 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:22.539361 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:22.539361 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:22.539361 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:22.539361 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:22.539361 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:22.539361 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:22.539361 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:32:22.539361 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:32:22.539361 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:32:22.539361 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:32:22.540431 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:22.540431 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:22.540431 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:22.540431 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:22.540431 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:22.540431 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:22.540431 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:22.540431 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:22.540431 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:22.540431 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:22.540431 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:22.540431 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:22.540431 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:22.540431 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:22.540431 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:22.540431 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:22.540431 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:32:22.541345 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:32:22.541345 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:32:22.541345 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:22.541345 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 473/968] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:32:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 335 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:32:22.543048 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6f45c4-6bec-41b2-b754-c26498b28451 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:22.543048 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6f45c4-6bec-41b2-b754-c26498b28451 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:22.543048 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6f45c4-6bec-41b2-b754-c26498b28451 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:22.544113 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 474/969] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:32:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:32:22.549052 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6f45c4-6bec-41b2-b754-c26498b28451 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:22.551332 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17a81bf3-6248-42cf-982f-85c9d9170b94 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:22.555041 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17a81bf3-6248-42cf-982f-85c9d9170b94 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:22.567649 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-17a81bf3-6248-42cf-982f-85c9d9170b94 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Oct 28 09:32:22.567649 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:22.567649 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:22.567649 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:22.567649 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:22.567649 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:22.567649 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:22.567649 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:22.567649 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:22.567649 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:22.567649 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:22.567649 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:22.567649 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 09:32:22.567649 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 09:32:22.567649 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 09:32:22.567649 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 09:32:22.569415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:22.569415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:22.569415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 09:32:22.569415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:32:22.569415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 09:32:22.569415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:32:22.569415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:22.569415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:22.569415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:22.569415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:22.569415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:22.569415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:22.569415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:22.569415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:22.569415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:22.569415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:22.569415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:22.569415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:22.577550 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:22.577550 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:22.577550 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 09:32:22.577550 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 09:32:22.577550 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 09:32:22.577550 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 09:32:22.577550 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 09:32:22.577550 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 09:32:22.577550 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Oct 28 09:32:22.577550 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:22.577550 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 496/970] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:32:22 2022] GET /identity/v3/users/cinder => generated 84 bytes in 286 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:22.577550 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc3055d-4313-4b3d-b887-26d0019cc8bd None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:22.577550 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc3055d-4313-4b3d-b887-26d0019cc8bd None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:22.579020 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc3055d-4313-4b3d-b887-26d0019cc8bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:22.591324 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-ccc3055d-4313-4b3d-b887-26d0019cc8bd None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:32:22.593169 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ccc3055d-4313-4b3d-b887-26d0019cc8bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:32:22.600760 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:22.678904 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 497/971] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:32:22 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:32:22.680887 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc3055d-4313-4b3d-b887-26d0019cc8bd None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:22.719046 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6f45c4-6bec-41b2-b754-c26498b28451 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:22.782713 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec4f0eaa-7617-47cf-bc16-3663c02114d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:22.784398 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4f0eaa-7617-47cf-bc16-3663c02114d0 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:22.785280 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4f0eaa-7617-47cf-bc16-3663c02114d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:22.786037 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4f0eaa-7617-47cf-bc16-3663c02114d0 None admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:22.788085 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec4f0eaa-7617-47cf-bc16-3663c02114d0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:22.789091 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec4f0eaa-7617-47cf-bc16-3663c02114d0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:22.791904 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec4f0eaa-7617-47cf-bc16-3663c02114d0 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:22.819575 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 475/972] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:32:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:32:22.824273 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec4f0eaa-7617-47cf-bc16-3663c02114d0 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:22.886446 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:22.904433 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc3055d-4313-4b3d-b887-26d0019cc8bd None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:22.969472 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a358d4-bb7b-4139-8171-fab88a9282fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:22.970341 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a358d4-bb7b-4139-8171-fab88a9282fd None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:22.970921 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a358d4-bb7b-4139-8171-fab88a9282fd None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:22.971370 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4a358d4-bb7b-4139-8171-fab88a9282fd None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:22.972598 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4a358d4-bb7b-4139-8171-fab88a9282fd None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:22.973111 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4a358d4-bb7b-4139-8171-fab88a9282fd None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:22.976086 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4a358d4-bb7b-4139-8171-fab88a9282fd None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:22.996829 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:23.013838 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec4f0eaa-7617-47cf-bc16-3663c02114d0 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:23.034443 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 498/973] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Oct 28 09:32:22 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 346 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:23.034443 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a358d4-bb7b-4139-8171-fab88a9282fd None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:23.073408 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08eb404-ccaf-4423-bfbb-620679c8f4a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:23.074295 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08eb404-ccaf-4423-bfbb-620679c8f4a4 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:23.074695 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08eb404-ccaf-4423-bfbb-620679c8f4a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:23.075051 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08eb404-ccaf-4423-bfbb-620679c8f4a4 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:23.085014 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a08eb404-ccaf-4423-bfbb-620679c8f4a4 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:23.086822 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a08eb404-ccaf-4423-bfbb-620679c8f4a4 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:23.100169 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-a08eb404-ccaf-4423-bfbb-620679c8f4a4 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 28 09:32:23.100169 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:23.100169 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:23.100169 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:23.100169 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:23.100169 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:23.100169 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:23.100169 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:23.100169 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:23.100169 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:23.100169 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:23.100169 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:23.100169 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 09:32:23.100169 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 09:32:23.100169 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 09:32:23.100169 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 09:32:23.101222 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:23.101222 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:23.101222 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 09:32:23.101222 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:32:23.101222 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 09:32:23.101222 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:32:23.101222 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:23.101222 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:23.101222 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:23.101222 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:23.101222 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:23.101222 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:23.101222 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:23.101222 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:23.101222 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:23.101222 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:23.101222 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:23.101222 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:23.104043 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:23.104043 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:23.104043 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 09:32:23.104043 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 09:32:23.104043 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 09:32:23.104043 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 09:32:23.104043 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 09:32:23.104043 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 09:32:23.104043 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Oct 28 09:32:23.104043 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:23.104043 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 476/974] 10.222.0.25 () {56 vars in 1117 bytes} [Fri Oct 28 09:32:22 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 278 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:32:23.104043 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf326a1-f906-4445-bb3e-d19259380eb1 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:23.104043 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf326a1-f906-4445-bb3e-d19259380eb1 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:23.110246 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf326a1-f906-4445-bb3e-d19259380eb1 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:23.110246 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 477/975] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:32:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:32:23.110246 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf326a1-f906-4445-bb3e-d19259380eb1 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:23.217336 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4a358d4-bb7b-4139-8171-fab88a9282fd None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:23.236822 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf326a1-f906-4445-bb3e-d19259380eb1 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:23.276791 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c21e0e-d9b2-4eb4-a07c-57955653b4f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:23.277687 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c21e0e-d9b2-4eb4-a07c-57955653b4f7 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:23.278088 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c21e0e-d9b2-4eb4-a07c-57955653b4f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:23.278374 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c21e0e-d9b2-4eb4-a07c-57955653b4f7 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:23.279028 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73c21e0e-d9b2-4eb4-a07c-57955653b4f7 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:23.280738 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73c21e0e-d9b2-4eb4-a07c-57955653b4f7 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:23.291541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-73c21e0e-d9b2-4eb4-a07c-57955653b4f7 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 09:32:23.291541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:23.291541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 09:32:23.291541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 09:32:23.291541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 09:32:23.291541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 09:32:23.291541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 09:32:23.291541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:32:23.291541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 09:32:23.291541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:23.291541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 09:32:23.291541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:23.291541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:23.291541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:23.291541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:23.291541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:23.291541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:23.293672 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:23.293672 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:23.293672 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:23.293672 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:23.293672 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:23.293672 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:23.293672 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 09:32:23.293672 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 09:32:23.293672 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 09:32:23.293672 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 09:32:23.293672 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:23.293672 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:23.293672 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:23.293672 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:23.293672 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:23.293672 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:23.295495 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:23.295495 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:23.295495 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:23.295495 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:23.295495 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:23.295495 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:23.295495 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:23.295495 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:23.295495 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:23.295495 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:23.295495 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 09:32:23.295495 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:32:23.295495 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 09:32:23.295495 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:23.297050 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 499/976] 10.222.0.25 () {56 vars in 1126 bytes} [Fri Oct 28 09:32:23 2022] GET /identity/v3/services/object-store => generated 93 bytes in 271 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:23.298367 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbeef23-6202-410c-ae9f-dcd40ecc64fb None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:23.298918 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbeef23-6202-410c-ae9f-dcd40ecc64fb None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:23.299194 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbeef23-6202-410c-ae9f-dcd40ecc64fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:23.310498 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e395c4f9-4996-4892-8f26-b4152ebfabb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:23.311574 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e395c4f9-4996-4892-8f26-b4152ebfabb8 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:23.312071 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e395c4f9-4996-4892-8f26-b4152ebfabb8 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:23.312564 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e395c4f9-4996-4892-8f26-b4152ebfabb8 None admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:23.314805 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e395c4f9-4996-4892-8f26-b4152ebfabb8 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:23.315587 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e395c4f9-4996-4892-8f26-b4152ebfabb8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:23.318256 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e395c4f9-4996-4892-8f26-b4152ebfabb8 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:23.319508 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-5fbeef23-6202-410c-ae9f-dcd40ecc64fb None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:32:23.320742 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-5fbeef23-6202-410c-ae9f-dcd40ecc64fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:32:23.337706 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:23.349163 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 478/977] 10.222.0.25 () {56 vars in 1149 bytes} [Fri Oct 28 09:32:23 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:23.349163 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e395c4f9-4996-4892-8f26-b4152ebfabb8 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:23.356623 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:23.415667 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 500/978] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:32:23 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:32:23.416559 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbeef23-6202-410c-ae9f-dcd40ecc64fb None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:23.491423 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e395c4f9-4996-4892-8f26-b4152ebfabb8 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:23.529595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0283342a-9933-44e5-94e1-cdab7b0d8bdf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:23.530219 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0283342a-9933-44e5-94e1-cdab7b0d8bdf None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:23.530535 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0283342a-9933-44e5-94e1-cdab7b0d8bdf None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:23.530815 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0283342a-9933-44e5-94e1-cdab7b0d8bdf None admin] PATH_INFO: `/v3/services` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:23.531463 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0283342a-9933-44e5-94e1-cdab7b0d8bdf None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:23.531772 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0283342a-9933-44e5-94e1-cdab7b0d8bdf None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:23.533169 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0283342a-9933-44e5-94e1-cdab7b0d8bdf None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:23.539798 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 479/979] 10.222.0.25 () {56 vars in 1122 bytes} [Fri Oct 28 09:32:23 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:23.547566 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0283342a-9933-44e5-94e1-cdab7b0d8bdf None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:23.575701 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbeef23-6202-410c-ae9f-dcd40ecc64fb None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:23.623212 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee838821-f5b9-4256-baaf-6796ebd27f92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:23.624158 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee838821-f5b9-4256-baaf-6796ebd27f92 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:23.624576 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee838821-f5b9-4256-baaf-6796ebd27f92 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:23.624980 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee838821-f5b9-4256-baaf-6796ebd27f92 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:23.632054 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:23.651467 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:23.682930 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0283342a-9933-44e5-94e1-cdab7b0d8bdf None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:23.723119 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee838821-f5b9-4256-baaf-6796ebd27f92 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:23.725071 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee838821-f5b9-4256-baaf-6796ebd27f92 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:23.740436 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b543ff0e-cb28-4b45-b644-0e72be164612 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:23.741982 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b543ff0e-cb28-4b45-b644-0e72be164612 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:23.742428 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b543ff0e-cb28-4b45-b644-0e72be164612 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:23.742887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b543ff0e-cb28-4b45-b644-0e72be164612 None admin] PATH_INFO: `/v3/services` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:23.744011 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b543ff0e-cb28-4b45-b644-0e72be164612 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:23.744547 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b543ff0e-cb28-4b45-b644-0e72be164612 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:23.747053 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b543ff0e-cb28-4b45-b644-0e72be164612 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:23.755309 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 480/980] 10.222.0.25 () {56 vars in 1122 bytes} [Fri Oct 28 09:32:23 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:32:23.756782 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b543ff0e-cb28-4b45-b644-0e72be164612 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:23.834858 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 501/981] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:32:23 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 419 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 09:32:23.835734 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee838821-f5b9-4256-baaf-6796ebd27f92 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:23.872691 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b543ff0e-cb28-4b45-b644-0e72be164612 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:23.885511 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:23.911191 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e77c4b-8ef9-42a0-b602-a93425fb4412 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:23.911810 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e77c4b-8ef9-42a0-b602-a93425fb4412 None admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:23.912115 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e77c4b-8ef9-42a0-b602-a93425fb4412 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:23.912419 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e77c4b-8ef9-42a0-b602-a93425fb4412 None admin] PATH_INFO: `/v3/services` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:23.913099 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e77c4b-8ef9-42a0-b602-a93425fb4412 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:23.914520 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e77c4b-8ef9-42a0-b602-a93425fb4412 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:23.918250 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:23.934127 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 481/982] 10.222.0.25 () {60 vars in 1141 bytes} [Fri Oct 28 09:32:23 2022] POST /identity/v3/services => generated 242 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:32:23.935417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e77c4b-8ef9-42a0-b602-a93425fb4412 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:23.960725 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee838821-f5b9-4256-baaf-6796ebd27f92 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:24.036556 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3cc994c-b82c-4e3f-b913-decd9b22654e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:24.037485 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3cc994c-b82c-4e3f-b913-decd9b22654e None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:24.037902 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3cc994c-b82c-4e3f-b913-decd9b22654e None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:24.038352 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3cc994c-b82c-4e3f-b913-decd9b22654e None admin] PATH_INFO: `/v3/endpoints` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:24.039668 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3cc994c-b82c-4e3f-b913-decd9b22654e None admin] RBAC: Adding query filter params (interface=admin, service_id=84ddeb9619a8482796ab05b4a3cce595, region_id=RegionOne) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:24.040270 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3cc994c-b82c-4e3f-b913-decd9b22654e None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:24.042585 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3cc994c-b82c-4e3f-b913-decd9b22654e None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:24.052213 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 502/983] 10.222.0.25 () {56 vars in 1249 bytes} [Fri Oct 28 09:32:23 2022] GET /identity/v3/endpoints?service_id=84ddeb9619a8482796ab05b4a3cce595&interface=admin®ion_id=RegionOne => generated 193 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:24.059168 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3cc994c-b82c-4e3f-b913-decd9b22654e None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:24.109118 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e77c4b-8ef9-42a0-b602-a93425fb4412 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:24.162986 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-481ce79a-41df-4562-8327-04fcf8ed46ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:24.163851 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481ce79a-41df-4562-8327-04fcf8ed46ad None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:24.164244 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481ce79a-41df-4562-8327-04fcf8ed46ad None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:24.164617 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481ce79a-41df-4562-8327-04fcf8ed46ad None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:24.182156 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-481ce79a-41df-4562-8327-04fcf8ed46ad None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:24.184728 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-481ce79a-41df-4562-8327-04fcf8ed46ad None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:24.210693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-481ce79a-41df-4562-8327-04fcf8ed46ad None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:32:24.210693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:24.210693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:24.210693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:24.210693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:24.210693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:24.210693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:24.210693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:24.210693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:24.210693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:24.210693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:24.210693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:24.210693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:32:24.210693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:32:24.210693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:32:24.210693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:32:24.211742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:24.211742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:24.211742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:24.211742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:24.211742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:24.211742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:24.211742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:24.211742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:24.211742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:24.211742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:24.211742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:24.211742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:24.211742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:24.211742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:24.211742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:24.211742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:24.211742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:32:24.212679 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:32:24.212679 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:32:24.212679 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:24.212679 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 482/984] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:32:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 278 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:24.219032 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3cc994c-b82c-4e3f-b913-decd9b22654e None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:24.220652 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-481ce79a-41df-4562-8327-04fcf8ed46ad None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:24.223470 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:24.266132 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6a2bc9-e9bc-41f3-b83f-e9b5308c5f4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:24.267194 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6a2bc9-e9bc-41f3-b83f-e9b5308c5f4b None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:24.267663 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6a2bc9-e9bc-41f3-b83f-e9b5308c5f4b None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:24.268131 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6a2bc9-e9bc-41f3-b83f-e9b5308c5f4b None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:24.269055 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6a2bc9-e9bc-41f3-b83f-e9b5308c5f4b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:24.270689 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6a2bc9-e9bc-41f3-b83f-e9b5308c5f4b None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:24.279745 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 503/985] 10.222.0.25 () {56 vars in 1087 bytes} [Fri Oct 28 09:32:24 2022] GET /identity/v3/services => generated 1727 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:32:24.290083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:24.368165 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-481ce79a-41df-4562-8327-04fcf8ed46ad None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:24.413012 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c739940-d7da-43ac-8c13-dafe1eb28cfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:24.413523 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c739940-d7da-43ac-8c13-dafe1eb28cfa None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:24.413798 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c739940-d7da-43ac-8c13-dafe1eb28cfa None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:24.414088 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c739940-d7da-43ac-8c13-dafe1eb28cfa None admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:24.414764 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c739940-d7da-43ac-8c13-dafe1eb28cfa None admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:24.415065 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c739940-d7da-43ac-8c13-dafe1eb28cfa None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:24.416505 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c739940-d7da-43ac-8c13-dafe1eb28cfa None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:24.432019 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 483/986] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:32:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:32:24.438351 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6a2bc9-e9bc-41f3-b83f-e9b5308c5f4b None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:24.610036 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6a2bc9-e9bc-41f3-b83f-e9b5308c5f4b None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:24.667277 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0c66d9-ed5f-4adf-b907-8168c069bb20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:24.668144 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0c66d9-ed5f-4adf-b907-8168c069bb20 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:24.668423 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0c66d9-ed5f-4adf-b907-8168c069bb20 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:24.668688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0c66d9-ed5f-4adf-b907-8168c069bb20 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:24.677096 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c0c66d9-ed5f-4adf-b907-8168c069bb20 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:24.679109 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c0c66d9-ed5f-4adf-b907-8168c069bb20 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:24.692288 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-7c0c66d9-ed5f-4adf-b907-8168c069bb20 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:32:24.692288 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:24.692288 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:24.692288 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:24.692288 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:24.692288 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:24.692288 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:24.692288 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:24.692288 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:24.692288 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:24.692288 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:24.692288 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:24.692288 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 09:32:24.692288 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 09:32:24.692288 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 09:32:24.694538 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 09:32:24.694538 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:24.694538 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:24.694538 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:24.694538 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:24.694538 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:24.694538 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:24.694538 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:24.694538 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:24.694538 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:24.694538 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:24.694538 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:24.694538 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:24.694538 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:24.694538 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:24.694538 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:24.694538 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:24.696516 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 09:32:24.696516 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 09:32:24.696516 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 09:32:24.696516 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 09:32:24.696516 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 09:32:24.696516 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 09:32:24.696516 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:32:24.696516 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:24.697969 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 504/987] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:32:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 256 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:24.700068 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c739940-d7da-43ac-8c13-dafe1eb28cfa None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:24.819968 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c739940-d7da-43ac-8c13-dafe1eb28cfa None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:24.830285 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:24.862482 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25fe2776-c9a8-446f-a377-ab6d76e252b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:24.863100 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fe2776-c9a8-446f-a377-ab6d76e252b8 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:24.863380 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fe2776-c9a8-446f-a377-ab6d76e252b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:24.863651 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fe2776-c9a8-446f-a377-ab6d76e252b8 None admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:24.864334 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25fe2776-c9a8-446f-a377-ab6d76e252b8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:24.864687 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25fe2776-c9a8-446f-a377-ab6d76e252b8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:24.866173 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25fe2776-c9a8-446f-a377-ab6d76e252b8 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:24.879930 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 484/988] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Oct 28 09:32:24 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:24.887005 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0c66d9-ed5f-4adf-b907-8168c069bb20 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:24.941194 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:25.075113 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0c66d9-ed5f-4adf-b907-8168c069bb20 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:25.137854 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-812b1152-5a60-47e3-8fa2-73676bf02e3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:25.138838 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812b1152-5a60-47e3-8fa2-73676bf02e3a None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:25.139329 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812b1152-5a60-47e3-8fa2-73676bf02e3a None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:25.139797 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812b1152-5a60-47e3-8fa2-73676bf02e3a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:25.141078 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-812b1152-5a60-47e3-8fa2-73676bf02e3a None admin] RBAC: Adding query filter params (role.id=4e19a68eecb04cb998e6aa7adfb48ad3, scope.project.id=455d1c826b974bc38497d18a516867c7, user.id=97c8877c74194a6bbbf25b3c8f5b7dbd) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:25.141609 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-812b1152-5a60-47e3-8fa2-73676bf02e3a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:25.144413 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-812b1152-5a60-47e3-8fa2-73676bf02e3a None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:25.156499 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 505/989] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:32:24 2022] GET /identity/v3/role_assignments?user.id=97c8877c74194a6bbbf25b3c8f5b7dbd&scope.project.id=455d1c826b974bc38497d18a516867c7&role.id=4e19a68eecb04cb998e6aa7adfb48ad3 => generated 608 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:26.216311 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d40a5a-4591-4d0c-886d-a5ca8a44e442 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:26.217043 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d40a5a-4591-4d0c-886d-a5ca8a44e442 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:26.217043 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d40a5a-4591-4d0c-886d-a5ca8a44e442 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:26.218591 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 485/990] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:32:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 09:32:26.224452 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc9bcac-86f9-41c3-bacd-e7d56281fe92 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:26.225250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc9bcac-86f9-41c3-bacd-e7d56281fe92 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:26.225250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc9bcac-86f9-41c3-bacd-e7d56281fe92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:26.246024 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-bdc9bcac-86f9-41c3-bacd-e7d56281fe92 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:32:26.247290 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-bdc9bcac-86f9-41c3-bacd-e7d56281fe92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:32:26.329422 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 506/991] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:32:26 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:32:26.602350 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d40a5a-4591-4d0c-886d-a5ca8a44e442 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:26.727798 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d40a5a-4591-4d0c-886d-a5ca8a44e442 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:26.750306 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4cb348-67be-4a61-8f1f-8281509254f5 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:26.750633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4cb348-67be-4a61-8f1f-8281509254f5 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:26.750960 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4cb348-67be-4a61-8f1f-8281509254f5 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:26.752380 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 507/992] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:32:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:32:26.758785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc22d868-7350-40dd-bc55-25b4298a2d70 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:26.759041 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc22d868-7350-40dd-bc55-25b4298a2d70 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:26.759370 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc22d868-7350-40dd-bc55-25b4298a2d70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:26.770486 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992d2594-07a8-40fa-b876-daad7ad84ed4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:26.771153 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992d2594-07a8-40fa-b876-daad7ad84ed4 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:26.771434 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992d2594-07a8-40fa-b876-daad7ad84ed4 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:26.772242 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992d2594-07a8-40fa-b876-daad7ad84ed4 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:26.772851 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-992d2594-07a8-40fa-b876-daad7ad84ed4 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:26.774287 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-992d2594-07a8-40fa-b876-daad7ad84ed4 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:26.781742 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-cc22d868-7350-40dd-bc55-25b4298a2d70 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:32:26.783015 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-cc22d868-7350-40dd-bc55-25b4298a2d70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:32:26.789555 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-992d2594-07a8-40fa-b876-daad7ad84ed4 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 28 09:32:26.789555 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:26.789555 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 09:32:26.789555 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 09:32:26.789555 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 09:32:26.789555 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 09:32:26.789555 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 09:32:26.789555 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:32:26.789555 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 28 09:32:26.789555 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:26.789555 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 09:32:26.789555 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:26.789555 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:26.789555 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:26.789555 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:26.789555 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:26.789555 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:26.791611 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:26.791611 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:26.791611 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:26.791611 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:26.791611 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:26.791611 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:26.791611 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 09:32:26.791611 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 09:32:26.791611 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 09:32:26.791611 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 09:32:26.791611 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:26.791611 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:26.791611 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:26.791611 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:26.791611 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:26.791611 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:26.793668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:26.793668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:26.793668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:26.793668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:26.793668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:26.793668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:26.793668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:26.793668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:26.793668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:26.793668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:26.793668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 09:32:26.793668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:32:26.793668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 28 09:32:26.793668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:26.793668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 486/993] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:32:26 2022] GET /identity/v3/services/compute => generated 88 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:32:26.797816 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992d2594-07a8-40fa-b876-daad7ad84ed4 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:26.808737 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:26.842601 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:26.869227 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 508/994] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:32:26 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 09:32:26.967140 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992d2594-07a8-40fa-b876-daad7ad84ed4 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:27.026533 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b03750-1813-4fcf-8747-785cc7d0f69d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:27.027561 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b03750-1813-4fcf-8747-785cc7d0f69d None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:27.028012 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b03750-1813-4fcf-8747-785cc7d0f69d None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:27.028454 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b03750-1813-4fcf-8747-785cc7d0f69d None admin] PATH_INFO: `/v3/services` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:27.029492 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88b03750-1813-4fcf-8747-785cc7d0f69d None admin] RBAC: Adding query filter params (name=compute) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:27.030026 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88b03750-1813-4fcf-8747-785cc7d0f69d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:27.032599 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88b03750-1813-4fcf-8747-785cc7d0f69d None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:27.046462 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 487/995] 10.222.0.25 () {56 vars in 1112 bytes} [Fri Oct 28 09:32:26 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:27.047896 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc22d868-7350-40dd-bc55-25b4298a2d70 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:27.165121 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc22d868-7350-40dd-bc55-25b4298a2d70 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:27.205254 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57a061c-9c63-4a80-bd37-83bcb235d254 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:27.206038 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57a061c-9c63-4a80-bd37-83bcb235d254 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:27.206483 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57a061c-9c63-4a80-bd37-83bcb235d254 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:27.206910 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57a061c-9c63-4a80-bd37-83bcb235d254 None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:27.207706 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b57a061c-9c63-4a80-bd37-83bcb235d254 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:27.208134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b57a061c-9c63-4a80-bd37-83bcb235d254 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:27.209954 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b57a061c-9c63-4a80-bd37-83bcb235d254 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:27.218956 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 509/996] 10.222.0.25 () {56 vars in 1112 bytes} [Fri Oct 28 09:32:27 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:27.223335 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b03750-1813-4fcf-8747-785cc7d0f69d None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:27.243904 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57a061c-9c63-4a80-bd37-83bcb235d254 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:27.275345 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:27.356487 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:27.364911 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b03750-1813-4fcf-8747-785cc7d0f69d None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:27.388738 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57a061c-9c63-4a80-bd37-83bcb235d254 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:27.409554 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aab356f3-1366-4e13-a4b4-1bbdcb86e615 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:27.410195 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab356f3-1366-4e13-a4b4-1bbdcb86e615 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:27.410586 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab356f3-1366-4e13-a4b4-1bbdcb86e615 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:27.410866 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab356f3-1366-4e13-a4b4-1bbdcb86e615 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:27.411590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aab356f3-1366-4e13-a4b4-1bbdcb86e615 None admin] RBAC: Adding query filter params (interface=public, service_id=55e1dee5e92d4a6dae5a78cda77160e9, region_id=RegionOne) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:27.411900 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aab356f3-1366-4e13-a4b4-1bbdcb86e615 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:27.414576 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aab356f3-1366-4e13-a4b4-1bbdcb86e615 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:27.430057 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 488/997] 10.222.0.25 () {56 vars in 1251 bytes} [Fri Oct 28 09:32:27 2022] GET /identity/v3/endpoints?service_id=55e1dee5e92d4a6dae5a78cda77160e9&interface=public®ion_id=RegionOne => generated 194 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:27.431785 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aab356f3-1366-4e13-a4b4-1bbdcb86e615 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:27.432458 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b58a088-bdad-4fa8-a210-ec550e141562 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:27.433292 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b58a088-bdad-4fa8-a210-ec550e141562 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:27.433689 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b58a088-bdad-4fa8-a210-ec550e141562 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:27.434099 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b58a088-bdad-4fa8-a210-ec550e141562 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:27.434928 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b58a088-bdad-4fa8-a210-ec550e141562 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:27.436515 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b58a088-bdad-4fa8-a210-ec550e141562 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:27.447362 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-2b58a088-bdad-4fa8-a210-ec550e141562 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 09:32:27.447362 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:27.447362 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 09:32:27.447362 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 09:32:27.447362 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 09:32:27.447362 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 09:32:27.447362 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 09:32:27.447362 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:32:27.447362 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 09:32:27.447362 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:27.447362 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 09:32:27.447362 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:27.447362 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:27.447362 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:27.447362 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:27.447362 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:27.447362 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:27.449323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:27.449323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:27.449323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:27.449323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:27.449323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:27.449323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:27.449323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 09:32:27.449323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 09:32:27.449323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 09:32:27.449323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 09:32:27.449323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:27.449323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:27.449323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:27.449323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:27.449323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:27.449323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:27.450440 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:27.450440 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:27.450440 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:27.450440 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:27.450440 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:27.450440 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:27.450440 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:27.450440 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:27.450440 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:27.450440 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:27.450440 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 09:32:27.450440 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:32:27.450440 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Oct 28 09:32:27.450440 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:27.450440 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 510/998] 10.222.0.25 () {56 vars in 1126 bytes} [Fri Oct 28 09:32:27 2022] GET /identity/v3/services/object-store => generated 93 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:32:27.455403 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b58a088-bdad-4fa8-a210-ec550e141562 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:27.561735 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aab356f3-1366-4e13-a4b4-1bbdcb86e615 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:27.581233 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:27.604704 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296f7e70-e6bc-4465-aa13-1c95bbeb0781 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:27.605317 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296f7e70-e6bc-4465-aa13-1c95bbeb0781 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:27.605589 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296f7e70-e6bc-4465-aa13-1c95bbeb0781 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:27.605870 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296f7e70-e6bc-4465-aa13-1c95bbeb0781 None admin] PATH_INFO: `/v3/services` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:27.606681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-296f7e70-e6bc-4465-aa13-1c95bbeb0781 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:27.607922 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-296f7e70-e6bc-4465-aa13-1c95bbeb0781 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:27.610241 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b58a088-bdad-4fa8-a210-ec550e141562 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:27.616012 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 489/999] 10.222.0.25 () {56 vars in 1087 bytes} [Fri Oct 28 09:32:27 2022] GET /identity/v3/services => generated 1727 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:32:27.658902 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3196861b-646f-497f-908f-56707051c095 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:27.659713 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3196861b-646f-497f-908f-56707051c095 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:27.660126 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3196861b-646f-497f-908f-56707051c095 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:27.660550 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3196861b-646f-497f-908f-56707051c095 None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:27.661358 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3196861b-646f-497f-908f-56707051c095 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:27.661810 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3196861b-646f-497f-908f-56707051c095 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:27.663524 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3196861b-646f-497f-908f-56707051c095 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:27.669972 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 511/1000] 10.222.0.25 () {56 vars in 1122 bytes} [Fri Oct 28 09:32:27 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:32:27.676204 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296f7e70-e6bc-4465-aa13-1c95bbeb0781 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:27.707696 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:27.809104 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296f7e70-e6bc-4465-aa13-1c95bbeb0781 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:27.853915 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844d910d-2aae-4719-9639-0ec9466cd1cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:27.854758 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844d910d-2aae-4719-9639-0ec9466cd1cc None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:27.855179 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844d910d-2aae-4719-9639-0ec9466cd1cc None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:27.855605 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844d910d-2aae-4719-9639-0ec9466cd1cc None admin] PATH_INFO: `/v3/services` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:27.856386 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-844d910d-2aae-4719-9639-0ec9466cd1cc None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:27.856823 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-844d910d-2aae-4719-9639-0ec9466cd1cc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:27.858486 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-844d910d-2aae-4719-9639-0ec9466cd1cc None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:27.864611 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 490/1001] 10.222.0.25 () {56 vars in 1122 bytes} [Fri Oct 28 09:32:27 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:32:27.871130 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3196861b-646f-497f-908f-56707051c095 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:27.997553 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3196861b-646f-497f-908f-56707051c095 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:28.039797 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46126510-9606-4051-bad2-fe8b4cc663d1 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:28.040086 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46126510-9606-4051-bad2-fe8b4cc663d1 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:28.040354 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46126510-9606-4051-bad2-fe8b4cc663d1 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:28.041977 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 491/1002] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:32:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 09:32:28.045513 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc53415-0f59-4d9c-966d-01136dfe06a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:28.046233 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc53415-0f59-4d9c-966d-01136dfe06a6 None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:28.046577 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc53415-0f59-4d9c-966d-01136dfe06a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:28.046853 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc53415-0f59-4d9c-966d-01136dfe06a6 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:28.047558 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc53415-0f59-4d9c-966d-01136dfe06a6 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:28.047712 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49c41cb-af4e-41c8-8486-bd0b1cdffc9d None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:28.047968 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49c41cb-af4e-41c8-8486-bd0b1cdffc9d None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:28.048214 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49c41cb-af4e-41c8-8486-bd0b1cdffc9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:28.048941 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc53415-0f59-4d9c-966d-01136dfe06a6 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:28.069305 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-a49c41cb-af4e-41c8-8486-bd0b1cdffc9d None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:32:28.070577 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-a49c41cb-af4e-41c8-8486-bd0b1cdffc9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:32:28.077958 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 512/1003] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:32:27 2022] POST /identity/v3/endpoints => generated 325 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:32:28.169463 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 492/1004] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:32:28 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:32:28.447747 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc53415-0f59-4d9c-966d-01136dfe06a6 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:28.451943 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:28.593213 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc53415-0f59-4d9c-966d-01136dfe06a6 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:28.653250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962c750d-b77e-4f65-8662-5d78a118bf26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:28.654306 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962c750d-b77e-4f65-8662-5d78a118bf26 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:28.654828 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962c750d-b77e-4f65-8662-5d78a118bf26 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:28.655303 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962c750d-b77e-4f65-8662-5d78a118bf26 None admin] PATH_INFO: `/v3/services/image` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:28.656335 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-962c750d-b77e-4f65-8662-5d78a118bf26 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:28.659127 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-962c750d-b77e-4f65-8662-5d78a118bf26 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:28.676986 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-962c750d-b77e-4f65-8662-5d78a118bf26 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 28 09:32:28.676986 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:28.676986 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 09:32:28.676986 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 09:32:28.676986 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 09:32:28.676986 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 09:32:28.676986 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 09:32:28.676986 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:32:28.676986 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 28 09:32:28.676986 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:28.676986 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 09:32:28.676986 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:28.676986 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:28.676986 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:28.676986 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:28.676986 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:28.676986 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:28.678158 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:28.678158 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:28.678158 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:28.678158 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:28.678158 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:28.678158 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:28.678158 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 09:32:28.678158 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 09:32:28.678158 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 09:32:28.678158 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 09:32:28.678158 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:28.678158 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:28.678158 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:28.678158 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:28.678158 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:28.678158 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:28.679078 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:28.679078 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:28.679078 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:28.679078 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:28.679078 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:28.679078 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:28.679078 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:28.679078 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:28.679078 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:28.679078 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:28.679078 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 09:32:28.679078 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:32:28.679078 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 28 09:32:28.679078 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:28.683728 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 513/1005] 10.222.0.25 () {56 vars in 1105 bytes} [Fri Oct 28 09:32:28 2022] GET /identity/v3/services/image => generated 86 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:28.687330 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49c41cb-af4e-41c8-8486-bd0b1cdffc9d None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:28.712956 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:28.862770 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49c41cb-af4e-41c8-8486-bd0b1cdffc9d None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:28.918918 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c3bc76-f93a-49ea-892c-28c3150e2b67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:28.919921 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c3bc76-f93a-49ea-892c-28c3150e2b67 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:28.920408 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c3bc76-f93a-49ea-892c-28c3150e2b67 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:28.920891 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c3bc76-f93a-49ea-892c-28c3150e2b67 None admin] PATH_INFO: `/v3/services` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:28.922011 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87c3bc76-f93a-49ea-892c-28c3150e2b67 None admin] RBAC: Adding query filter params (name=image) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:28.922575 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87c3bc76-f93a-49ea-892c-28c3150e2b67 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:28.925031 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87c3bc76-f93a-49ea-892c-28c3150e2b67 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:28.938666 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 493/1006] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:32:28 2022] GET /identity/v3/services?name=image => generated 122 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:32:28.941165 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962c750d-b77e-4f65-8662-5d78a118bf26 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:29.054306 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:29.114873 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962c750d-b77e-4f65-8662-5d78a118bf26 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:29.174152 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-942c60ab-a2ad-44b4-a7ae-7f979073eaab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:29.175369 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942c60ab-a2ad-44b4-a7ae-7f979073eaab None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:29.175956 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942c60ab-a2ad-44b4-a7ae-7f979073eaab None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:29.176544 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-942c60ab-a2ad-44b4-a7ae-7f979073eaab None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:29.177741 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942c60ab-a2ad-44b4-a7ae-7f979073eaab None admin] RBAC: Adding query filter params (type=image) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:29.178779 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942c60ab-a2ad-44b4-a7ae-7f979073eaab None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:29.180189 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-942c60ab-a2ad-44b4-a7ae-7f979073eaab None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:29.187896 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 514/1007] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:32:28 2022] GET /identity/v3/services?type=image => generated 122 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:32:30.002472 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6123685c-daaa-4394-b29c-881c4710032b None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:30.002983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6123685c-daaa-4394-b29c-881c4710032b None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:30.003313 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6123685c-daaa-4394-b29c-881c4710032b None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:30.005909 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 494/1008] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:32:29 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:32:30.012094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f73dd1-e7b3-45cb-bf1a-87e67de4a158 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:30.012879 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f73dd1-e7b3-45cb-bf1a-87e67de4a158 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:30.012879 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88f73dd1-e7b3-45cb-bf1a-87e67de4a158 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:30.032965 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-88f73dd1-e7b3-45cb-bf1a-87e67de4a158 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:32:30.034834 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-88f73dd1-e7b3-45cb-bf1a-87e67de4a158 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:32:30.115587 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 515/1009] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:32:30 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 09:32:30.386501 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6123685c-daaa-4394-b29c-881c4710032b None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:30.530858 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:30.575418 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6123685c-daaa-4394-b29c-881c4710032b None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:30.616921 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa665e75-e803-409b-9522-bb264f567879 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:30.617567 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa665e75-e803-409b-9522-bb264f567879 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:30.617844 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa665e75-e803-409b-9522-bb264f567879 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:30.618133 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa665e75-e803-409b-9522-bb264f567879 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:30.618760 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa665e75-e803-409b-9522-bb264f567879 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:30.620138 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa665e75-e803-409b-9522-bb264f567879 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:30.630083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-aa665e75-e803-409b-9522-bb264f567879 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Oct 28 09:32:30.630083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:30.630083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 09:32:30.630083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 09:32:30.630083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 09:32:30.630083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 09:32:30.630083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 09:32:30.630083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:32:30.630083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 28 09:32:30.630083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:30.630083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 09:32:30.630083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:30.630083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:30.630083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:30.630083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:30.630083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:30.630083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:30.632340 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:30.632340 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:30.632340 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:30.632340 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:30.632340 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:30.632340 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:30.632340 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 09:32:30.632340 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 09:32:30.632340 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 09:32:30.632340 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 09:32:30.632340 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:30.632340 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:30.632340 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:30.632340 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:30.632340 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:30.632340 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:30.634066 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:30.634066 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:30.634066 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:30.634066 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:30.634066 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:30.634066 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:30.634066 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:30.634066 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:30.634066 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:30.634066 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:30.634066 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 09:32:30.634066 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:32:30.634066 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Oct 28 09:32:30.634066 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:30.636644 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 495/1010] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:32:30 2022] GET /identity/v3/services/compute => generated 88 bytes in 251 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:30.637817 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f73dd1-e7b3-45cb-bf1a-87e67de4a158 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:30.646015 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2c9d65-f1b3-4b53-99c1-6ef6bce2da00 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:30.647254 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2c9d65-f1b3-4b53-99c1-6ef6bce2da00 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:30.648452 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2c9d65-f1b3-4b53-99c1-6ef6bce2da00 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:30.652105 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 496/1011] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:32:30 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:32:30.660094 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae06a26-63d7-401c-ac47-b3936a4e0ea0 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:30.660589 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae06a26-63d7-401c-ac47-b3936a4e0ea0 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:30.661008 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae06a26-63d7-401c-ac47-b3936a4e0ea0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:30.682477 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-2ae06a26-63d7-401c-ac47-b3936a4e0ea0 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:32:30.683667 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-2ae06a26-63d7-401c-ac47-b3936a4e0ea0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:32:30.714566 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:30.763081 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88f73dd1-e7b3-45cb-bf1a-87e67de4a158 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:30.766465 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 497/1012] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:32:30 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 09:32:30.806329 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a21fbd-3cd4-4338-8072-567a8e8709e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:30.806960 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a21fbd-3cd4-4338-8072-567a8e8709e7 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:30.807242 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a21fbd-3cd4-4338-8072-567a8e8709e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:30.807495 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a21fbd-3cd4-4338-8072-567a8e8709e7 None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:30.808179 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55a21fbd-3cd4-4338-8072-567a8e8709e7 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:30.808491 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55a21fbd-3cd4-4338-8072-567a8e8709e7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:30.809818 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55a21fbd-3cd4-4338-8072-567a8e8709e7 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:30.815811 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 516/1013] 10.222.0.25 () {56 vars in 1112 bytes} [Fri Oct 28 09:32:30 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:30.821567 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae06a26-63d7-401c-ac47-b3936a4e0ea0 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:30.900309 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:30.937905 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae06a26-63d7-401c-ac47-b3936a4e0ea0 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:30.978722 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3feffb-5262-40c7-a232-0ee5be2e2ab5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:30.979336 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3feffb-5262-40c7-a232-0ee5be2e2ab5 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:30.979616 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3feffb-5262-40c7-a232-0ee5be2e2ab5 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:30.979889 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3feffb-5262-40c7-a232-0ee5be2e2ab5 None admin] PATH_INFO: `/v3/services` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:30.980563 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d3feffb-5262-40c7-a232-0ee5be2e2ab5 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:30.980866 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d3feffb-5262-40c7-a232-0ee5be2e2ab5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:30.982237 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d3feffb-5262-40c7-a232-0ee5be2e2ab5 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:30.994864 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 498/1014] 10.222.0.25 () {56 vars in 1112 bytes} [Fri Oct 28 09:32:30 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:32:30.995899 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a21fbd-3cd4-4338-8072-567a8e8709e7 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:31.033589 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3feffb-5262-40c7-a232-0ee5be2e2ab5 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:31.129712 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a21fbd-3cd4-4338-8072-567a8e8709e7 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:31.148994 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3feffb-5262-40c7-a232-0ee5be2e2ab5 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:31.158811 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:31.171750 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e720ac6a-74aa-4718-b7c0-81f9f8ffbda3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:31.172535 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e720ac6a-74aa-4718-b7c0-81f9f8ffbda3 None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:31.172994 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e720ac6a-74aa-4718-b7c0-81f9f8ffbda3 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:31.173453 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e720ac6a-74aa-4718-b7c0-81f9f8ffbda3 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:31.174195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e720ac6a-74aa-4718-b7c0-81f9f8ffbda3 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:31.175787 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e720ac6a-74aa-4718-b7c0-81f9f8ffbda3 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:31.189927 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd1cc3e5-bbee-43b2-8cec-a3590e2a74fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:31.190866 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1cc3e5-bbee-43b2-8cec-a3590e2a74fe None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:31.191144 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1cc3e5-bbee-43b2-8cec-a3590e2a74fe None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:31.191421 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1cc3e5-bbee-43b2-8cec-a3590e2a74fe None admin] PATH_INFO: `/v3/domains/default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:31.192735 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd1cc3e5-bbee-43b2-8cec-a3590e2a74fe None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:31.194843 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd1cc3e5-bbee-43b2-8cec-a3590e2a74fe None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:31.196694 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 499/1015] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:32:31 2022] GET /identity/v3/domains/default => generated 199 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:32:31.210424 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 517/1016] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:32:30 2022] POST /identity/v3/endpoints => generated 334 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:32:31.210650 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd1cc3e5-bbee-43b2-8cec-a3590e2a74fe None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:31.214361 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:31.383321 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd1cc3e5-bbee-43b2-8cec-a3590e2a74fe None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:31.449915 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9872b051-410a-4efe-8f10-529551e277ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:31.450499 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9872b051-410a-4efe-8f10-529551e277ac None admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:31.450780 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9872b051-410a-4efe-8f10-529551e277ac None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:31.451052 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9872b051-410a-4efe-8f10-529551e277ac None admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:31.451715 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9872b051-410a-4efe-8f10-529551e277ac None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:31.453138 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9872b051-410a-4efe-8f10-529551e277ac None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:31.470671 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba6767b-ac84-41fa-baa7-cadfeb5c8427 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:31.470959 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba6767b-ac84-41fa-baa7-cadfeb5c8427 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:31.471241 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba6767b-ac84-41fa-baa7-cadfeb5c8427 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:31.472592 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 518/1017] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:32:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 09:32:31.477961 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db4a8f44-c6c8-4543-a51a-ad602b1a0f48 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:31.478329 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db4a8f44-c6c8-4543-a51a-ad602b1a0f48 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:31.478642 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db4a8f44-c6c8-4543-a51a-ad602b1a0f48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:31.492625 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 500/1018] 10.222.0.25 () {60 vars in 1141 bytes} [Fri Oct 28 09:32:31 2022] POST /identity/v3/projects => generated 311 bytes in 291 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:32:31.503429 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-db4a8f44-c6c8-4543-a51a-ad602b1a0f48 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:32:31.504674 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-db4a8f44-c6c8-4543-a51a-ad602b1a0f48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:32:31.630465 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 519/1019] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:32:31 2022] POST /identity/v3/auth/tokens => generated 1873 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:32:31.897520 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9872b051-410a-4efe-8f10-529551e277ac None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:31.949517 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:32.003107 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9872b051-410a-4efe-8f10-529551e277ac None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:32.040166 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6914b377-9bac-4499-a44a-7e6f5b7faeb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:32.040862 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6914b377-9bac-4499-a44a-7e6f5b7faeb1 None admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:32.041103 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6914b377-9bac-4499-a44a-7e6f5b7faeb1 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:32.041317 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6914b377-9bac-4499-a44a-7e6f5b7faeb1 None admin] PATH_INFO: `/v3/services` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:32.042002 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6914b377-9bac-4499-a44a-7e6f5b7faeb1 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:32.043402 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6914b377-9bac-4499-a44a-7e6f5b7faeb1 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:32.060485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 501/1020] 10.222.0.25 () {60 vars in 1141 bytes} [Fri Oct 28 09:32:31 2022] POST /identity/v3/services => generated 242 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:32:33.530036 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b32af01-7e32-40c9-b9b5-365affe79910 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:33.530721 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b32af01-7e32-40c9-b9b5-365affe79910 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:33.530721 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b32af01-7e32-40c9-b9b5-365affe79910 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:33.532131 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 520/1021] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:32:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:32:33.537682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0970f4e4-0ede-4dd0-a317-66db805175d4 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:33.538124 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0970f4e4-0ede-4dd0-a317-66db805175d4 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:33.538236 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0970f4e4-0ede-4dd0-a317-66db805175d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:33.557582 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-0970f4e4-0ede-4dd0-a317-66db805175d4 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:32:33.558866 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-0970f4e4-0ede-4dd0-a317-66db805175d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:32:33.642977 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 502/1022] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:32:33 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:32:33.723340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3123a4-755f-48b9-a096-fa3146172e26 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:33.723627 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3123a4-755f-48b9-a096-fa3146172e26 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:33.724082 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3123a4-755f-48b9-a096-fa3146172e26 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:33.725648 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 521/1023] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:32:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:32:33.731126 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae5a937-9bf1-4299-b23e-0f4ab76cfaf8 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:33.731410 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae5a937-9bf1-4299-b23e-0f4ab76cfaf8 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:33.731682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae5a937-9bf1-4299-b23e-0f4ab76cfaf8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:33.750765 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-7ae5a937-9bf1-4299-b23e-0f4ab76cfaf8 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:32:33.751972 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-7ae5a937-9bf1-4299-b23e-0f4ab76cfaf8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:32:33.782934 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:33.826442 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 503/1024] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:32:33 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:32:33.915466 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d3123a4-755f-48b9-a096-fa3146172e26 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:34.022297 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d3123a4-755f-48b9-a096-fa3146172e26 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:34.036352 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:34.059244 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f12d6c-038b-4c85-bf8b-2feab945c5b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:34.059883 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f12d6c-038b-4c85-bf8b-2feab945c5b3 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:34.060167 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f12d6c-038b-4c85-bf8b-2feab945c5b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:34.060445 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3f12d6c-038b-4c85-bf8b-2feab945c5b3 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:34.071902 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3f12d6c-038b-4c85-bf8b-2feab945c5b3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:32:34.076772 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-a3f12d6c-038b-4c85-bf8b-2feab945c5b3 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 28 09:32:34.076772 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:34.076772 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:34.076772 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:34.076772 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:34.076772 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:34.076772 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:34.076772 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:34.076772 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:34.076772 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:34.076772 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:34.076772 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:34.076772 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:32:34.076772 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:32:34.076772 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:32:34.076772 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:32:34.077899 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:32:34.077899 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:32:34.077899 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:32:34.077899 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:32:34.077899 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:34.077899 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:34.077899 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:34.077899 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:34.077899 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:34.077899 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:34.077899 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:34.077899 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:34.077899 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:34.077899 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:34.077899 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:34.077899 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:34.083805 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:34.083805 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:34.083805 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:34.083805 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:34.083805 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:32:34.083805 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:32:34.083805 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:32:34.083805 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:32:34.083805 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:32:34.083805 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:32:34.083805 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 28 09:32:34.083805 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:34.083805 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 522/1025] 10.222.0.25 () {56 vars in 1120 bytes} [Fri Oct 28 09:32:33 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:34.084778 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae5a937-9bf1-4299-b23e-0f4ab76cfaf8 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:34.097361 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f12d6c-038b-4c85-bf8b-2feab945c5b3 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:34.201630 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae5a937-9bf1-4299-b23e-0f4ab76cfaf8 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:34.213468 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3f12d6c-038b-4c85-bf8b-2feab945c5b3 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:34.239557 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b45f199-4526-400d-9b6e-b976ba7bf925 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:34.240184 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b45f199-4526-400d-9b6e-b976ba7bf925 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:34.241039 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b45f199-4526-400d-9b6e-b976ba7bf925 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:34.241312 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b45f199-4526-400d-9b6e-b976ba7bf925 None admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:34.241988 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b45f199-4526-400d-9b6e-b976ba7bf925 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:34.242299 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b45f199-4526-400d-9b6e-b976ba7bf925 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:34.243716 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b45f199-4526-400d-9b6e-b976ba7bf925 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:34.251430 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df13a87-467c-44d5-889a-2c812107ab37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:34.252072 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df13a87-467c-44d5-889a-2c812107ab37 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:34.257510 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df13a87-467c-44d5-889a-2c812107ab37 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:34.257694 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 504/1026] 10.222.0.25 () {56 vars in 1115 bytes} [Fri Oct 28 09:32:34 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:34.257807 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df13a87-467c-44d5-889a-2c812107ab37 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:34.257807 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df13a87-467c-44d5-889a-2c812107ab37 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:34.257807 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df13a87-467c-44d5-889a-2c812107ab37 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:34.257807 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 523/1027] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:32:34 2022] GET /identity/v3/domains/default => generated 199 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:34.262144 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df13a87-467c-44d5-889a-2c812107ab37 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:34.262278 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b45f199-4526-400d-9b6e-b976ba7bf925 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:34.293065 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:34.307561 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:34.367902 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b45f199-4526-400d-9b6e-b976ba7bf925 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:34.368379 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df13a87-467c-44d5-889a-2c812107ab37 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:34.402616 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25c640d-dc8e-4acf-a528-9de77a0c8d73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:34.403248 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25c640d-dc8e-4acf-a528-9de77a0c8d73 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:34.403527 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25c640d-dc8e-4acf-a528-9de77a0c8d73 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:34.403783 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b9b83e-5f3c-4f7f-9d28-7d6c8d725a9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:34.403894 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25c640d-dc8e-4acf-a528-9de77a0c8d73 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:34.404385 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b9b83e-5f3c-4f7f-9d28-7d6c8d725a9f None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:34.404664 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b9b83e-5f3c-4f7f-9d28-7d6c8d725a9f None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:34.404936 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b9b83e-5f3c-4f7f-9d28-7d6c8d725a9f None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:34.405595 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b9b83e-5f3c-4f7f-9d28-7d6c8d725a9f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:34.407043 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b9b83e-5f3c-4f7f-9d28-7d6c8d725a9f None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:34.415180 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c25c640d-dc8e-4acf-a528-9de77a0c8d73 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:34.416688 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c25c640d-dc8e-4acf-a528-9de77a0c8d73 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:34.429375 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 524/1028] 10.222.0.25 () {60 vars in 1132 bytes} [Fri Oct 28 09:32:34 2022] POST /identity/v3/users => generated 285 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:32:34.430729 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836c9ca0-74d3-495e-b573-3bed7c747c5f None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:34.431015 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836c9ca0-74d3-495e-b573-3bed7c747c5f None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:34.431287 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836c9ca0-74d3-495e-b573-3bed7c747c5f None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:34.432900 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 525/1029] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:32:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:32:34.433085 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-c25c640d-dc8e-4acf-a528-9de77a0c8d73 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:32:34.433085 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:34.433085 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:34.433085 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:34.433085 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:34.433085 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:34.433085 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:34.433085 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:34.433085 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:34.433085 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:34.433085 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:34.433085 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:34.433085 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:32:34.433085 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:32:34.433085 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:32:34.433085 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:32:34.434078 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:34.434078 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:34.434078 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:34.434078 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:34.434078 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:34.434078 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:34.434078 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:34.434078 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:34.434078 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:34.434078 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:34.434078 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:34.434078 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:34.434078 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:34.434078 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:34.434078 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:34.434078 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:34.434078 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:32:34.435044 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:32:34.435044 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:32:34.435044 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:34.435044 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 505/1030] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:32:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:34.438890 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d866d4-33ba-4e62-9371-62e6b9c42fe1 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:34.439444 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d866d4-33ba-4e62-9371-62e6b9c42fe1 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:34.439659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d866d4-33ba-4e62-9371-62e6b9c42fe1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:34.440790 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25c640d-dc8e-4acf-a528-9de77a0c8d73 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:34.461207 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-91d866d4-33ba-4e62-9371-62e6b9c42fe1 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:32:34.462552 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-91d866d4-33ba-4e62-9371-62e6b9c42fe1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:32:34.541106 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:34.554518 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 526/1031] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:32:34 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:32:34.580896 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25c640d-dc8e-4acf-a528-9de77a0c8d73 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:34.621667 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f35b69b-e8d0-47d3-b0c8-ee9f59546ec1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:34.622649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f35b69b-e8d0-47d3-b0c8-ee9f59546ec1 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:34.623134 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f35b69b-e8d0-47d3-b0c8-ee9f59546ec1 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:34.623615 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f35b69b-e8d0-47d3-b0c8-ee9f59546ec1 None admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:34.624285 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f35b69b-e8d0-47d3-b0c8-ee9f59546ec1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:34.624594 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f35b69b-e8d0-47d3-b0c8-ee9f59546ec1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:34.625988 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f35b69b-e8d0-47d3-b0c8-ee9f59546ec1 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:34.639722 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 506/1032] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:32:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:34.646181 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d866d4-33ba-4e62-9371-62e6b9c42fe1 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:34.675744 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:34.760381 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d866d4-33ba-4e62-9371-62e6b9c42fe1 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:34.795864 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5e02fc-7636-4720-8a32-1adeb2494cbc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:34.796503 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5e02fc-7636-4720-8a32-1adeb2494cbc None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:34.796787 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5e02fc-7636-4720-8a32-1adeb2494cbc None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:34.797060 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5e02fc-7636-4720-8a32-1adeb2494cbc None admin] PATH_INFO: `/v3/users/nova` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:34.804646 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd5e02fc-7636-4720-8a32-1adeb2494cbc None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:34.806008 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd5e02fc-7636-4720-8a32-1adeb2494cbc None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:34.818912 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-dd5e02fc-7636-4720-8a32-1adeb2494cbc None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 28 09:32:34.818912 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:34.818912 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:34.818912 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:34.818912 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:34.818912 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:34.818912 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:34.818912 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:34.818912 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:34.818912 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:34.818912 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:34.818912 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:34.818912 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 09:32:34.818912 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 09:32:34.818912 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 09:32:34.818912 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 09:32:34.819925 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:34.819925 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:34.819925 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 09:32:34.819925 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:32:34.819925 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 09:32:34.819925 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:32:34.819925 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:34.819925 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:34.819925 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:34.819925 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:34.819925 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:34.819925 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:34.819925 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:34.819925 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:34.819925 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:34.819925 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:34.819925 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:34.819925 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:34.820903 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:34.820903 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:34.820903 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 09:32:34.820903 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 09:32:34.820903 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 09:32:34.820903 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 09:32:34.820903 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 09:32:34.820903 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 09:32:34.820903 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 28 09:32:34.820903 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:34.820903 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 527/1033] 10.222.0.25 () {56 vars in 1093 bytes} [Fri Oct 28 09:32:34 2022] GET /identity/v3/users/nova => generated 82 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:34.823691 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f35b69b-e8d0-47d3-b0c8-ee9f59546ec1 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:34.828764 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5e02fc-7636-4720-8a32-1adeb2494cbc None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:34.923629 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:34.937593 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f35b69b-e8d0-47d3-b0c8-ee9f59546ec1 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:34.940952 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5e02fc-7636-4720-8a32-1adeb2494cbc None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:34.972335 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9881dd78-9e1d-4a4f-b654-46c7361d6df3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:34.973033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9881dd78-9e1d-4a4f-b654-46c7361d6df3 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:34.973316 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9881dd78-9e1d-4a4f-b654-46c7361d6df3 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:34.973594 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9881dd78-9e1d-4a4f-b654-46c7361d6df3 None admin] PATH_INFO: `/v3/services/image` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:34.974187 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9881dd78-9e1d-4a4f-b654-46c7361d6df3 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:34.975732 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9881dd78-9e1d-4a4f-b654-46c7361d6df3 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:34.978555 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-546222c3-3d56-447a-b3d0-72edbfe155af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:34.979351 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546222c3-3d56-447a-b3d0-72edbfe155af None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:34.979801 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546222c3-3d56-447a-b3d0-72edbfe155af None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:34.980226 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546222c3-3d56-447a-b3d0-72edbfe155af None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:34.981118 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-546222c3-3d56-447a-b3d0-72edbfe155af None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:34.981601 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-546222c3-3d56-447a-b3d0-72edbfe155af None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:34.983174 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-546222c3-3d56-447a-b3d0-72edbfe155af None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:34.984649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-9881dd78-9e1d-4a4f-b654-46c7361d6df3 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 28 09:32:34.984649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:34.984649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 09:32:34.984649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 09:32:34.984649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 09:32:34.984649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 09:32:34.984649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 09:32:34.984649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:32:34.984649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 28 09:32:34.984649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:34.984649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 09:32:34.984649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:34.984649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:34.984649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:34.984649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:34.984649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:34.984649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:34.985875 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:34.985875 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:34.985875 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:34.985875 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:34.985875 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:34.985875 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:34.985875 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 09:32:34.985875 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 09:32:34.985875 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 09:32:34.985875 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 09:32:34.985875 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:34.985875 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:34.985875 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:34.985875 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:34.985875 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:34.985875 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:34.986915 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:34.986915 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:34.986915 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:34.986915 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:34.986915 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:34.986915 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:34.986915 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:34.986915 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:34.986915 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:34.986915 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:34.986915 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 09:32:34.986915 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:32:34.986915 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 28 09:32:34.986915 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:34.986915 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 507/1034] 10.222.0.25 () {56 vars in 1105 bytes} [Fri Oct 28 09:32:34 2022] GET /identity/v3/services/image => generated 86 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:32:34.992267 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9881dd78-9e1d-4a4f-b654-46c7361d6df3 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:34.996039 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:35.008063 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 528/1035] 10.222.0.25 () {56 vars in 1133 bytes} [Fri Oct 28 09:32:34 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:35.013698 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-546222c3-3d56-447a-b3d0-72edbfe155af None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:35.106293 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9881dd78-9e1d-4a4f-b654-46c7361d6df3 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:35.127529 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-546222c3-3d56-447a-b3d0-72edbfe155af None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:35.142909 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0365f23d-8e52-4d85-a6e5-07ff539eeaa0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:35.143548 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0365f23d-8e52-4d85-a6e5-07ff539eeaa0 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:35.143832 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0365f23d-8e52-4d85-a6e5-07ff539eeaa0 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:35.144108 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0365f23d-8e52-4d85-a6e5-07ff539eeaa0 None admin] PATH_INFO: `/v3/services` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:35.144752 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0365f23d-8e52-4d85-a6e5-07ff539eeaa0 None admin] RBAC: Adding query filter params (name=image) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:35.145065 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0365f23d-8e52-4d85-a6e5-07ff539eeaa0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:35.146506 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0365f23d-8e52-4d85-a6e5-07ff539eeaa0 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:35.152184 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 508/1036] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:32:34 2022] GET /identity/v3/services?name=image => generated 122 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:32:35.157835 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0365f23d-8e52-4d85-a6e5-07ff539eeaa0 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:35.170832 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d7f451-4330-4557-b812-b1e4fc0e52fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:35.171461 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d7f451-4330-4557-b812-b1e4fc0e52fc None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:35.171738 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d7f451-4330-4557-b812-b1e4fc0e52fc None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:35.172016 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d7f451-4330-4557-b812-b1e4fc0e52fc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:35.187694 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:35.232695 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:35.247976 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d7f451-4330-4557-b812-b1e4fc0e52fc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:35.249442 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81d7f451-4330-4557-b812-b1e4fc0e52fc None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:35.268628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0365f23d-8e52-4d85-a6e5-07ff539eeaa0 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:35.304959 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55cfaa43-e8f9-48f6-a978-9b7d6df256af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:35.305584 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55cfaa43-e8f9-48f6-a978-9b7d6df256af None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:35.305868 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55cfaa43-e8f9-48f6-a978-9b7d6df256af None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:35.306140 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55cfaa43-e8f9-48f6-a978-9b7d6df256af None admin] PATH_INFO: `/v3/services` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:35.306810 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55cfaa43-e8f9-48f6-a978-9b7d6df256af None admin] RBAC: Adding query filter params (type=image) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:35.307118 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55cfaa43-e8f9-48f6-a978-9b7d6df256af None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:35.308502 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55cfaa43-e8f9-48f6-a978-9b7d6df256af None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:35.318167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 509/1037] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:32:35 2022] GET /identity/v3/services?type=image => generated 350 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:32:35.319341 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55cfaa43-e8f9-48f6-a978-9b7d6df256af None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:35.367701 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 529/1038] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:32:35 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 354 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 28 09:32:35.373351 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d7f451-4330-4557-b812-b1e4fc0e52fc None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:35.426782 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:35.432869 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55cfaa43-e8f9-48f6-a978-9b7d6df256af None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:35.469352 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:35.476566 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e955532-6526-4dea-a234-99cfff99a9a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:35.477173 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e955532-6526-4dea-a234-99cfff99a9a9 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:35.477450 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e955532-6526-4dea-a234-99cfff99a9a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:35.477724 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e955532-6526-4dea-a234-99cfff99a9a9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:35.478451 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e955532-6526-4dea-a234-99cfff99a9a9 None admin] RBAC: Adding query filter params (interface=public, service_id=78c5babff3a743e394d81cd6a5749c59, region_id=RegionOne) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:35.478760 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e955532-6526-4dea-a234-99cfff99a9a9 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:35.480123 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e955532-6526-4dea-a234-99cfff99a9a9 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:35.488227 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d7f451-4330-4557-b812-b1e4fc0e52fc None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:35.490047 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 510/1039] 10.222.0.25 () {56 vars in 1251 bytes} [Fri Oct 28 09:32:35 2022] GET /identity/v3/endpoints?service_id=78c5babff3a743e394d81cd6a5749c59&interface=public®ion_id=RegionOne => generated 194 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:35.491328 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e955532-6526-4dea-a234-99cfff99a9a9 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:35.526403 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a4014f-20f8-4bef-8791-2bedfcf03e3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:35.527084 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a4014f-20f8-4bef-8791-2bedfcf03e3a None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:35.527365 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a4014f-20f8-4bef-8791-2bedfcf03e3a None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:35.527639 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a4014f-20f8-4bef-8791-2bedfcf03e3a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:35.540177 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a4014f-20f8-4bef-8791-2bedfcf03e3a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:35.541645 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a4014f-20f8-4bef-8791-2bedfcf03e3a None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:35.561913 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-34a4014f-20f8-4bef-8791-2bedfcf03e3a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:32:35.561913 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:35.561913 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:35.561913 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:35.561913 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:35.561913 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:35.561913 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:35.561913 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:35.561913 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:35.561913 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:35.561913 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:35.561913 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:35.561913 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:32:35.561913 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:32:35.561913 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:32:35.561913 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:32:35.568997 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:35.568997 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:35.568997 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:35.568997 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:35.568997 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:35.568997 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:35.568997 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:35.568997 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:35.568997 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:35.568997 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:35.568997 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:35.568997 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:35.568997 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:35.568997 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:35.568997 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:35.568997 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:35.568997 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:32:35.569953 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:32:35.569953 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:32:35.569953 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:35.569953 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 530/1040] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:32:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:35.569953 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a4014f-20f8-4bef-8791-2bedfcf03e3a None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:35.611948 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e955532-6526-4dea-a234-99cfff99a9a9 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:35.648991 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04324494-46d0-4bec-9aad-0d03474c82e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:35.649518 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04324494-46d0-4bec-9aad-0d03474c82e2 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:35.649799 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04324494-46d0-4bec-9aad-0d03474c82e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:35.650073 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04324494-46d0-4bec-9aad-0d03474c82e2 None admin] PATH_INFO: `/v3/services` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:35.650721 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04324494-46d0-4bec-9aad-0d03474c82e2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:35.652087 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04324494-46d0-4bec-9aad-0d03474c82e2 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:35.670430 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 511/1041] 10.222.0.25 () {56 vars in 1087 bytes} [Fri Oct 28 09:32:35 2022] GET /identity/v3/services => generated 1957 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:32:35.685859 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:35.692876 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a4014f-20f8-4bef-8791-2bedfcf03e3a None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:35.730555 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2f5060-4a22-4a10-9df3-0f3ba09b836e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:35.731089 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2f5060-4a22-4a10-9df3-0f3ba09b836e None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:35.731447 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2f5060-4a22-4a10-9df3-0f3ba09b836e None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:35.731713 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2f5060-4a22-4a10-9df3-0f3ba09b836e None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:35.732324 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb2f5060-4a22-4a10-9df3-0f3ba09b836e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:35.732665 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb2f5060-4a22-4a10-9df3-0f3ba09b836e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:35.733986 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb2f5060-4a22-4a10-9df3-0f3ba09b836e None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:35.748315 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 531/1042] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:32:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:35.758062 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04324494-46d0-4bec-9aad-0d03474c82e2 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:35.875773 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:35.927652 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04324494-46d0-4bec-9aad-0d03474c82e2 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:35.962039 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee4b7039-a9e8-4336-a451-34ce4db84a04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:35.962691 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4b7039-a9e8-4336-a451-34ce4db84a04 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:35.962970 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4b7039-a9e8-4336-a451-34ce4db84a04 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:35.963242 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4b7039-a9e8-4336-a451-34ce4db84a04 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:35.969903 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee4b7039-a9e8-4336-a451-34ce4db84a04 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:35.971362 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee4b7039-a9e8-4336-a451-34ce4db84a04 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:35.982624 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-ee4b7039-a9e8-4336-a451-34ce4db84a04 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:32:35.982624 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:35.982624 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:35.982624 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:35.982624 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:35.982624 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:35.982624 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:35.982624 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:35.982624 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:35.982624 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:35.982624 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:35.982624 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:35.982624 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 09:32:35.982624 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 09:32:35.982624 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 09:32:35.983563 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 09:32:35.983563 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:35.983563 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:35.983563 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:35.983563 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:35.983563 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:35.983563 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:35.983563 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:35.983563 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:35.983563 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:35.983563 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:35.983563 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:35.983563 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:35.983563 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:35.983563 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:35.983563 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:35.983563 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:35.984524 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 09:32:35.984524 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 09:32:35.984524 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 09:32:35.984524 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 09:32:35.984524 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 09:32:35.984524 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 09:32:35.984524 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:32:35.984524 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:35.984524 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 512/1043] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:32:35 2022] GET /identity/v3/projects/service => generated 88 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:35.991688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2f5060-4a22-4a10-9df3-0f3ba09b836e None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:36.092477 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2f5060-4a22-4a10-9df3-0f3ba09b836e None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:36.127323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f1e46c-94a8-40b3-9438-ede24a328a0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:36.127940 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f1e46c-94a8-40b3-9438-ede24a328a0a None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:36.128217 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f1e46c-94a8-40b3-9438-ede24a328a0a None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:36.128489 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f1e46c-94a8-40b3-9438-ede24a328a0a None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:36.129256 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4f1e46c-94a8-40b3-9438-ede24a328a0a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:36.129598 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4f1e46c-94a8-40b3-9438-ede24a328a0a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:36.131555 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4f1e46c-94a8-40b3-9438-ede24a328a0a None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:36.148460 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 532/1044] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Oct 28 09:32:35 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:36.150092 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee4b7039-a9e8-4336-a451-34ce4db84a04 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:36.256962 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee4b7039-a9e8-4336-a451-34ce4db84a04 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:36.277153 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:36.293753 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d94be2f-cb2d-40ce-b4ff-19cfa7259496 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:36.294458 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d94be2f-cb2d-40ce-b4ff-19cfa7259496 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:36.294749 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d94be2f-cb2d-40ce-b4ff-19cfa7259496 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:36.295026 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d94be2f-cb2d-40ce-b4ff-19cfa7259496 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:36.295765 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d94be2f-cb2d-40ce-b4ff-19cfa7259496 None admin] RBAC: Adding query filter params (role.id=4e19a68eecb04cb998e6aa7adfb48ad3, scope.project.id=455d1c826b974bc38497d18a516867c7, user.id=688ceb201d304c2da5373c3cf36c2d79) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:36.296078 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d94be2f-cb2d-40ce-b4ff-19cfa7259496 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:36.297522 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d94be2f-cb2d-40ce-b4ff-19cfa7259496 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:36.310428 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 513/1045] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:32:36 2022] GET /identity/v3/role_assignments?user.id=688ceb201d304c2da5373c3cf36c2d79&scope.project.id=455d1c826b974bc38497d18a516867c7&role.id=4e19a68eecb04cb998e6aa7adfb48ad3 => generated 259 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:36.704218 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e06fb5b-f975-4455-8f69-b2c973630036 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:36.704790 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e06fb5b-f975-4455-8f69-b2c973630036 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:36.704790 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e06fb5b-f975-4455-8f69-b2c973630036 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:36.706152 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 533/1046] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:32:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:32:36.712145 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28965e97-647a-40f4-972a-8842d28bc78e None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:36.712634 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28965e97-647a-40f4-972a-8842d28bc78e None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:36.712730 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28965e97-647a-40f4-972a-8842d28bc78e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:36.732455 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-28965e97-647a-40f4-972a-8842d28bc78e None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:32:36.733704 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-28965e97-647a-40f4-972a-8842d28bc78e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:32:36.809571 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 514/1047] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:32:36 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:32:37.082076 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e06fb5b-f975-4455-8f69-b2c973630036 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:37.101594 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:37.194585 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e06fb5b-f975-4455-8f69-b2c973630036 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:37.230632 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a1ae63-d8f5-4118-a9f3-244dee9e5843 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:37.231560 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a1ae63-d8f5-4118-a9f3-244dee9e5843 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:37.231991 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a1ae63-d8f5-4118-a9f3-244dee9e5843 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:37.232416 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1a1ae63-d8f5-4118-a9f3-244dee9e5843 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:37.243775 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1a1ae63-d8f5-4118-a9f3-244dee9e5843 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:32:37.248922 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-b1a1ae63-d8f5-4118-a9f3-244dee9e5843 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:32:37.248922 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:37.248922 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:37.248922 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:37.248922 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:37.248922 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:37.248922 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:37.248922 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:37.248922 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:37.248922 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:37.248922 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:37.248922 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:37.248922 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:32:37.248922 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:32:37.248922 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:32:37.248922 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:32:37.250063 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:32:37.250063 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:32:37.250063 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:32:37.250063 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:32:37.250063 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:37.250063 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:37.250063 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:37.250063 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:37.250063 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:37.250063 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:37.250063 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:37.250063 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:37.250063 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:37.250063 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:37.250063 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:37.250063 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:37.251220 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:37.251220 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:37.251220 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:37.251220 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:37.251220 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:32:37.251220 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:32:37.251220 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:32:37.251220 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:32:37.251220 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:32:37.251220 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:32:37.251220 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:32:37.251220 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:37.253595 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 534/1048] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Oct 28 09:32:37 2022] GET /identity/v3/roles/admin => generated 83 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:37.259391 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28965e97-647a-40f4-972a-8842d28bc78e None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:37.364913 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28965e97-647a-40f4-972a-8842d28bc78e None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:37.380766 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:37.406747 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cedb5eb-63f8-428d-9bdf-386412dae79d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:37.407864 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cedb5eb-63f8-428d-9bdf-386412dae79d None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:37.408605 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cedb5eb-63f8-428d-9bdf-386412dae79d None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:37.409090 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cedb5eb-63f8-428d-9bdf-386412dae79d None admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:37.409962 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cedb5eb-63f8-428d-9bdf-386412dae79d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:37.410475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cedb5eb-63f8-428d-9bdf-386412dae79d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:37.412050 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cedb5eb-63f8-428d-9bdf-386412dae79d None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:37.422283 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 515/1049] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:32:37 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:37.428720 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a1ae63-d8f5-4118-a9f3-244dee9e5843 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:37.510545 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:37.536416 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1a1ae63-d8f5-4118-a9f3-244dee9e5843 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:37.578765 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a7da3c-0a70-4b60-b4c4-81723de151c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:37.579467 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a7da3c-0a70-4b60-b4c4-81723de151c6 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:37.579749 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a7da3c-0a70-4b60-b4c4-81723de151c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:37.580027 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a7da3c-0a70-4b60-b4c4-81723de151c6 None admin] PATH_INFO: `/v3/users/253ac57d64d9408c8963a93085b43e7b` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:37.595600 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a7da3c-0a70-4b60-b4c4-81723de151c6 None admin] RBAC: Authorizing `identity:get_user(user_id=253ac57d64d9408c8963a93085b43e7b)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:37.597062 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a7da3c-0a70-4b60-b4c4-81723de151c6 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:37.603494 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 535/1050] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:32:37 2022] GET /identity/v3/users/253ac57d64d9408c8963a93085b43e7b => generated 285 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:37.604715 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cedb5eb-63f8-428d-9bdf-386412dae79d None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:37.711796 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cedb5eb-63f8-428d-9bdf-386412dae79d None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:37.746979 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b25f6ae-0630-40e5-a434-9875f9752df0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:37.747620 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b25f6ae-0630-40e5-a434-9875f9752df0 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:37.747865 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b25f6ae-0630-40e5-a434-9875f9752df0 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:37.748138 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b25f6ae-0630-40e5-a434-9875f9752df0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:37.796795 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:37.825307 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b25f6ae-0630-40e5-a434-9875f9752df0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:37.826816 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b25f6ae-0630-40e5-a434-9875f9752df0 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:37.916503 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03e1f25-407e-4bd3-a5c7-068e31e186e8 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:37.917060 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03e1f25-407e-4bd3-a5c7-068e31e186e8 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:37.917060 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03e1f25-407e-4bd3-a5c7-068e31e186e8 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:37.918543 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 536/1051] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:32:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 09:32:37.923823 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b939f04d-e219-482a-b2a1-8dd936a74507 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:37.924152 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b939f04d-e219-482a-b2a1-8dd936a74507 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:37.924387 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b939f04d-e219-482a-b2a1-8dd936a74507 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:37.931510 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 516/1052] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:32:37 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 28 09:32:37.937040 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b25f6ae-0630-40e5-a434-9875f9752df0 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:37.948604 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-b939f04d-e219-482a-b2a1-8dd936a74507 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:32:37.949778 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-b939f04d-e219-482a-b2a1-8dd936a74507 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:32:38.018493 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:38.025302 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 537/1053] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:32:37 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 09:32:38.044937 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b25f6ae-0630-40e5-a434-9875f9752df0 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:38.081275 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f77593dd-163a-4f74-ab1e-aaf2ca6db8cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:38.082287 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77593dd-163a-4f74-ab1e-aaf2ca6db8cd None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:38.082743 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77593dd-163a-4f74-ab1e-aaf2ca6db8cd None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:38.083193 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77593dd-163a-4f74-ab1e-aaf2ca6db8cd None admin] PATH_INFO: `/v3/projects/506899eec53d4333a54f3567f37027a4` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:38.085121 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f77593dd-163a-4f74-ab1e-aaf2ca6db8cd None admin] RBAC: Authorizing `identity:get_project(project_id=506899eec53d4333a54f3567f37027a4)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:38.086763 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f77593dd-163a-4f74-ab1e-aaf2ca6db8cd None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:38.092359 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 517/1054] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:32:37 2022] GET /identity/v3/projects/506899eec53d4333a54f3567f37027a4 => generated 311 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:32:38.095440 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b939f04d-e219-482a-b2a1-8dd936a74507 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:38.162458 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:38.201905 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b939f04d-e219-482a-b2a1-8dd936a74507 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:38.237945 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70719ede-33e0-4afa-9787-dfda581ae1f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:38.238583 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70719ede-33e0-4afa-9787-dfda581ae1f5 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:38.238863 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70719ede-33e0-4afa-9787-dfda581ae1f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:38.239134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70719ede-33e0-4afa-9787-dfda581ae1f5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:38.239856 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70719ede-33e0-4afa-9787-dfda581ae1f5 None admin] RBAC: Adding query filter params (role.id=e13d798a98ea4c3dbb87a236ec683d32, scope.project.id=506899eec53d4333a54f3567f37027a4, user.id=253ac57d64d9408c8963a93085b43e7b) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:38.240152 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70719ede-33e0-4afa-9787-dfda581ae1f5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:38.242089 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70719ede-33e0-4afa-9787-dfda581ae1f5 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:38.254666 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 538/1055] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:32:38 2022] GET /identity/v3/role_assignments?user.id=253ac57d64d9408c8963a93085b43e7b&scope.project.id=506899eec53d4333a54f3567f37027a4&role.id=e13d798a98ea4c3dbb87a236ec683d32 => generated 259 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:32:38.296825 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f77593dd-163a-4f74-ab1e-aaf2ca6db8cd None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:38.417408 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f77593dd-163a-4f74-ab1e-aaf2ca6db8cd None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:38.456854 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b71ccc3-1e11-46c2-97ca-f4036ffd2aa0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:38.457519 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b71ccc3-1e11-46c2-97ca-f4036ffd2aa0 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:38.457803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b71ccc3-1e11-46c2-97ca-f4036ffd2aa0 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:38.458080 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b71ccc3-1e11-46c2-97ca-f4036ffd2aa0 None admin] PATH_INFO: `/v3/services/image` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:38.458700 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b71ccc3-1e11-46c2-97ca-f4036ffd2aa0 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:38.460161 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b71ccc3-1e11-46c2-97ca-f4036ffd2aa0 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:38.469643 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-0b71ccc3-1e11-46c2-97ca-f4036ffd2aa0 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Oct 28 09:32:38.469643 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:38.469643 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 09:32:38.469643 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 09:32:38.469643 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 09:32:38.469643 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 09:32:38.469643 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 09:32:38.469643 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:32:38.469643 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 28 09:32:38.469643 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:38.469643 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 09:32:38.469643 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:38.469643 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:38.469643 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:38.469643 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:38.469643 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:38.469643 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:38.470778 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:38.470778 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:38.470778 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:38.470778 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:38.470778 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:38.470778 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:38.470778 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 09:32:38.470778 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 09:32:38.470778 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 09:32:38.470778 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 09:32:38.470778 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:38.470778 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:38.470778 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:38.470778 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:38.470778 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:38.470778 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:38.471620 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:38.471620 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:38.471620 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:38.471620 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:38.471620 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:38.471620 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:38.471620 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:38.471620 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:38.471620 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:38.471620 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:38.471620 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 09:32:38.471620 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:32:38.471620 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Oct 28 09:32:38.471620 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:38.474484 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 518/1056] 10.222.0.25 () {56 vars in 1105 bytes} [Fri Oct 28 09:32:38 2022] GET /identity/v3/services/image => generated 86 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:38.477345 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70719ede-33e0-4afa-9787-dfda581ae1f5 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:38.519099 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd00ab7-0160-48c2-9979-d21bc6ba789d None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:38.519438 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd00ab7-0160-48c2-9979-d21bc6ba789d None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:38.519659 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd00ab7-0160-48c2-9979-d21bc6ba789d None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:38.521207 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 519/1057] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:32:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 09:32:38.526831 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c56c78-5b49-4f56-b56b-0e3bba878982 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:38.527129 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c56c78-5b49-4f56-b56b-0e3bba878982 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:38.527416 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c56c78-5b49-4f56-b56b-0e3bba878982 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:38.547480 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-a3c56c78-5b49-4f56-b56b-0e3bba878982 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:32:38.548613 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-a3c56c78-5b49-4f56-b56b-0e3bba878982 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:32:38.550805 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:38.597360 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70719ede-33e0-4afa-9787-dfda581ae1f5 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:38.628358 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:38.637777 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 520/1058] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:32:38 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 09:32:38.643165 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ffce90-ca23-4493-b1c1-eeb5e1ad099b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:38.643800 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ffce90-ca23-4493-b1c1-eeb5e1ad099b None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:38.644083 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ffce90-ca23-4493-b1c1-eeb5e1ad099b None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:38.644358 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ffce90-ca23-4493-b1c1-eeb5e1ad099b None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:38.644995 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ffce90-ca23-4493-b1c1-eeb5e1ad099b None admin] RBAC: Adding query filter params (name=image) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:38.645299 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ffce90-ca23-4493-b1c1-eeb5e1ad099b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:38.646742 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ffce90-ca23-4493-b1c1-eeb5e1ad099b None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:38.657140 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 539/1059] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:32:38 2022] GET /identity/v3/services?name=image => generated 122 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:38.658163 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c56c78-5b49-4f56-b56b-0e3bba878982 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:38.770688 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c56c78-5b49-4f56-b56b-0e3bba878982 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:38.808842 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848de1da-fbd4-4f0b-9ab1-750f48a95504 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:38.809371 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848de1da-fbd4-4f0b-9ab1-750f48a95504 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:38.809650 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848de1da-fbd4-4f0b-9ab1-750f48a95504 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:38.809928 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848de1da-fbd4-4f0b-9ab1-750f48a95504 None admin] PATH_INFO: `/v3/services` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:38.810586 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-848de1da-fbd4-4f0b-9ab1-750f48a95504 None admin] RBAC: Adding query filter params (type=image) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:38.810891 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-848de1da-fbd4-4f0b-9ab1-750f48a95504 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:38.812259 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-848de1da-fbd4-4f0b-9ab1-750f48a95504 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:38.822222 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 521/1060] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:32:38 2022] GET /identity/v3/services?type=image => generated 350 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:38.824020 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ffce90-ca23-4493-b1c1-eeb5e1ad099b None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:38.909859 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848de1da-fbd4-4f0b-9ab1-750f48a95504 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:38.912856 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:38.936063 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ffce90-ca23-4493-b1c1-eeb5e1ad099b None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:38.973731 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28b3639-a734-400f-9181-13f93c54f7d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:38.974355 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28b3639-a734-400f-9181-13f93c54f7d6 None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:38.974742 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28b3639-a734-400f-9181-13f93c54f7d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:38.975022 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28b3639-a734-400f-9181-13f93c54f7d6 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:38.975711 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c28b3639-a734-400f-9181-13f93c54f7d6 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:38.977080 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c28b3639-a734-400f-9181-13f93c54f7d6 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:39.010432 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 540/1061] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:32:38 2022] POST /identity/v3/endpoints => generated 327 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:32:39.033224 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848de1da-fbd4-4f0b-9ab1-750f48a95504 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:39.095088 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a86484c6-976e-41c6-ae82-5e406154d13b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:39.095771 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86484c6-976e-41c6-ae82-5e406154d13b None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:39.096121 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86484c6-976e-41c6-ae82-5e406154d13b None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:39.096459 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a86484c6-976e-41c6-ae82-5e406154d13b None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:39.110199 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a86484c6-976e-41c6-ae82-5e406154d13b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:32:39.115488 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-a86484c6-976e-41c6-ae82-5e406154d13b None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 28 09:32:39.115488 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:39.115488 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:39.115488 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:39.115488 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:39.115488 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:39.115488 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:39.115488 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:39.115488 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:39.115488 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:39.115488 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:39.115488 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:39.115488 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:32:39.115488 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:32:39.115488 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:32:39.115488 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:32:39.116590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:32:39.116590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:32:39.116590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:32:39.116590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:32:39.116590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:39.116590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:39.116590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:39.116590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:39.116590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:39.116590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:39.116590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:39.116590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:39.116590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:39.116590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:39.116590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:39.116590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:39.119417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:39.119417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:39.119417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:39.119417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:39.119417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:32:39.119417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:32:39.119417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:32:39.119417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:32:39.119417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:32:39.119417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:32:39.119417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 28 09:32:39.119417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:39.119417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 522/1062] 10.222.0.25 () {56 vars in 1120 bytes} [Fri Oct 28 09:32:38 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:32:39.123152 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28b3639-a734-400f-9181-13f93c54f7d6 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:39.182750 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:39.243725 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28b3639-a734-400f-9181-13f93c54f7d6 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:39.281164 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235b36c3-e886-40af-b7ce-2dfefed7a842 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:39.281786 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235b36c3-e886-40af-b7ce-2dfefed7a842 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:39.282071 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235b36c3-e886-40af-b7ce-2dfefed7a842 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:39.282348 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235b36c3-e886-40af-b7ce-2dfefed7a842 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:39.283056 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-235b36c3-e886-40af-b7ce-2dfefed7a842 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:39.283365 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-235b36c3-e886-40af-b7ce-2dfefed7a842 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:39.284991 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-235b36c3-e886-40af-b7ce-2dfefed7a842 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:39.295368 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 541/1063] 10.222.0.25 () {56 vars in 1115 bytes} [Fri Oct 28 09:32:39 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:32:39.301213 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a86484c6-976e-41c6-ae82-5e406154d13b None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:39.366306 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:39.411025 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a86484c6-976e-41c6-ae82-5e406154d13b None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:39.457737 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0893700c-0ea4-47d6-9873-a301bc9442b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:39.458580 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0893700c-0ea4-47d6-9873-a301bc9442b7 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:39.459060 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0893700c-0ea4-47d6-9873-a301bc9442b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:39.459447 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0893700c-0ea4-47d6-9873-a301bc9442b7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:39.472357 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0893700c-0ea4-47d6-9873-a301bc9442b7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:39.474123 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0893700c-0ea4-47d6-9873-a301bc9442b7 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:39.492333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-0893700c-0ea4-47d6-9873-a301bc9442b7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:32:39.492333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:39.492333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:39.492333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:39.492333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:39.492333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:39.492333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:39.492333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:39.492333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:39.492333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:39.492333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:39.492333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:39.492333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:32:39.492333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:32:39.492333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:32:39.492333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:32:39.493552 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:39.493552 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:39.493552 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:39.493552 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:39.493552 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:39.493552 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:39.493552 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:39.493552 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:39.493552 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:39.493552 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:39.493552 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:39.493552 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:39.493552 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:39.493552 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:39.493552 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:39.493552 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:39.493552 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:32:39.494843 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:32:39.494843 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:32:39.494843 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:39.500049 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 523/1064] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:32:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:32:39.501284 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235b36c3-e886-40af-b7ce-2dfefed7a842 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:39.617804 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235b36c3-e886-40af-b7ce-2dfefed7a842 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:39.629955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:39.656034 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6235c9ec-7c35-4886-80a7-912b03288cdc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:39.656650 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6235c9ec-7c35-4886-80a7-912b03288cdc None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:39.656974 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6235c9ec-7c35-4886-80a7-912b03288cdc None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:39.657214 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6235c9ec-7c35-4886-80a7-912b03288cdc None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:39.657852 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6235c9ec-7c35-4886-80a7-912b03288cdc None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:39.658207 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6235c9ec-7c35-4886-80a7-912b03288cdc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:39.659799 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6235c9ec-7c35-4886-80a7-912b03288cdc None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:39.682128 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 542/1065] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:32:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:39.683592 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0893700c-0ea4-47d6-9873-a301bc9442b7 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:39.794560 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0893700c-0ea4-47d6-9873-a301bc9442b7 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:39.806098 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:39.832434 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d1203c-12fb-441d-ba02-0d668971ae21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:39.833330 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d1203c-12fb-441d-ba02-0d668971ae21 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:39.833775 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d1203c-12fb-441d-ba02-0d668971ae21 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:39.834732 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d1203c-12fb-441d-ba02-0d668971ae21 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:39.843350 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d1203c-12fb-441d-ba02-0d668971ae21 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:39.844996 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d1203c-12fb-441d-ba02-0d668971ae21 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:39.857039 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-12d1203c-12fb-441d-ba02-0d668971ae21 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 28 09:32:39.857039 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:39.857039 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:39.857039 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:39.857039 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:39.857039 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:39.857039 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:39.857039 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:39.857039 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:39.857039 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:39.857039 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:39.857039 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:39.857039 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 09:32:39.857039 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 09:32:39.857039 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 09:32:39.857039 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 09:32:39.858083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:39.858083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:39.858083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 09:32:39.858083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:32:39.858083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 09:32:39.858083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:32:39.858083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:39.858083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:39.858083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:39.858083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:39.858083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:39.858083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:39.858083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:39.858083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:39.858083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:39.858083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:39.858083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:39.858083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:39.859086 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:39.859086 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:39.859086 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 09:32:39.859086 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 09:32:39.859086 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 09:32:39.859086 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 09:32:39.859086 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 09:32:39.859086 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 09:32:39.859086 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 28 09:32:39.859086 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:39.859086 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 524/1066] 10.222.0.25 () {56 vars in 1093 bytes} [Fri Oct 28 09:32:39 2022] GET /identity/v3/users/nova => generated 82 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:39.866168 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6235c9ec-7c35-4886-80a7-912b03288cdc None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:39.980926 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6235c9ec-7c35-4886-80a7-912b03288cdc None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:40.016172 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d07e09-1b98-4c3d-a07b-eff599c77f11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:40.016781 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d07e09-1b98-4c3d-a07b-eff599c77f11 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:40.017067 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d07e09-1b98-4c3d-a07b-eff599c77f11 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:40.017337 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d07e09-1b98-4c3d-a07b-eff599c77f11 None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:40.018074 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d07e09-1b98-4c3d-a07b-eff599c77f11 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:40.018380 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d07e09-1b98-4c3d-a07b-eff599c77f11 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:40.019815 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d07e09-1b98-4c3d-a07b-eff599c77f11 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:40.043084 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 543/1067] 10.222.0.25 () {56 vars in 1133 bytes} [Fri Oct 28 09:32:39 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:40.044247 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d1203c-12fb-441d-ba02-0d668971ae21 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:40.147743 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d1203c-12fb-441d-ba02-0d668971ae21 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:40.184420 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d679ce-82a6-4d9d-9363-1aa604a24991 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:40.185091 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d679ce-82a6-4d9d-9363-1aa604a24991 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:40.185371 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d679ce-82a6-4d9d-9363-1aa604a24991 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:40.185649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d679ce-82a6-4d9d-9363-1aa604a24991 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:40.198409 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d679ce-82a6-4d9d-9363-1aa604a24991 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:40.199915 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d679ce-82a6-4d9d-9363-1aa604a24991 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:40.215642 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-31d679ce-82a6-4d9d-9363-1aa604a24991 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:32:40.215642 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:40.215642 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:40.215642 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:40.215642 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:40.215642 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:40.215642 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:40.215642 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:40.215642 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:40.215642 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:40.215642 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:40.215642 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:40.215642 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:32:40.215642 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:32:40.215642 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:32:40.215642 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:32:40.216704 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:40.216704 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:40.216704 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:40.216704 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:40.216704 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:40.216704 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:40.216704 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:40.216704 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:40.216704 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:40.216704 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:40.216704 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:40.216704 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:40.216704 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:40.216704 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:40.216704 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:40.216704 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:40.216704 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:32:40.217645 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:32:40.217645 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:32:40.217645 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:40.217645 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 525/1068] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:32:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:40.224142 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d07e09-1b98-4c3d-a07b-eff599c77f11 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:40.254790 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:40.333415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d07e09-1b98-4c3d-a07b-eff599c77f11 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:40.373701 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24574b10-672c-4670-9359-2922f9fe2b11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:40.374316 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24574b10-672c-4670-9359-2922f9fe2b11 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:40.374622 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24574b10-672c-4670-9359-2922f9fe2b11 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:40.374896 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24574b10-672c-4670-9359-2922f9fe2b11 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:40.375547 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24574b10-672c-4670-9359-2922f9fe2b11 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:40.375849 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24574b10-672c-4670-9359-2922f9fe2b11 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:40.377199 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24574b10-672c-4670-9359-2922f9fe2b11 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:40.393605 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 544/1069] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:32:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:40.394968 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d679ce-82a6-4d9d-9363-1aa604a24991 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:40.422314 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:40.457704 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf7c2fd-2925-4298-b682-853e7b871bc4 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:40.457995 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf7c2fd-2925-4298-b682-853e7b871bc4 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:40.458281 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf7c2fd-2925-4298-b682-853e7b871bc4 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:40.460324 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 545/1070] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:32:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 09:32:40.465256 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079614f0-7c6e-44ef-a3b4-1eb3c1bdecaa None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:40.465545 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079614f0-7c6e-44ef-a3b4-1eb3c1bdecaa None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:40.465879 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079614f0-7c6e-44ef-a3b4-1eb3c1bdecaa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:40.485818 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-079614f0-7c6e-44ef-a3b4-1eb3c1bdecaa None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:32:40.487064 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-079614f0-7c6e-44ef-a3b4-1eb3c1bdecaa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:32:40.507663 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d679ce-82a6-4d9d-9363-1aa604a24991 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:40.542515 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec2b03f9-84a6-4a10-9ada-96b69e362f7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:40.543158 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec2b03f9-84a6-4a10-9ada-96b69e362f7f None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:40.543440 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec2b03f9-84a6-4a10-9ada-96b69e362f7f None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:40.543727 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec2b03f9-84a6-4a10-9ada-96b69e362f7f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:40.550909 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec2b03f9-84a6-4a10-9ada-96b69e362f7f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:40.552382 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec2b03f9-84a6-4a10-9ada-96b69e362f7f None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:40.564238 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 546/1071] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:32:40 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 09:32:40.568326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-ec2b03f9-84a6-4a10-9ada-96b69e362f7f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:32:40.568326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:40.568326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:40.568326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:40.568326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:40.568326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:40.568326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:40.568326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:40.568326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:40.568326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:40.568326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:40.568326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:40.568326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 09:32:40.568326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 09:32:40.568326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 09:32:40.569298 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 09:32:40.569298 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:40.569298 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:40.569298 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:40.569298 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:40.569298 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:40.569298 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:40.569298 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:40.569298 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:40.569298 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:40.569298 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:40.569298 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:40.569298 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:40.569298 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:40.569298 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:40.569298 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:40.569298 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:40.570251 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 09:32:40.570251 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 09:32:40.570251 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 09:32:40.570251 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 09:32:40.570251 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 09:32:40.570251 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 09:32:40.570251 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:32:40.570251 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:40.576414 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 526/1072] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:32:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:40.577066 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079614f0-7c6e-44ef-a3b4-1eb3c1bdecaa None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:40.595322 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:40.682313 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079614f0-7c6e-44ef-a3b4-1eb3c1bdecaa None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:40.718757 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f6e4c3-a3d9-4272-8121-81836d017703 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:40.719377 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f6e4c3-a3d9-4272-8121-81836d017703 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:40.719663 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f6e4c3-a3d9-4272-8121-81836d017703 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:40.719970 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f6e4c3-a3d9-4272-8121-81836d017703 None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:40.720643 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f6e4c3-a3d9-4272-8121-81836d017703 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:40.720953 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f6e4c3-a3d9-4272-8121-81836d017703 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:40.722377 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f6e4c3-a3d9-4272-8121-81836d017703 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:40.740411 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 547/1073] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Oct 28 09:32:40 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:40.741754 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec2b03f9-84a6-4a10-9ada-96b69e362f7f None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:40.835925 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f6e4c3-a3d9-4272-8121-81836d017703 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:40.843932 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:40.866291 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec2b03f9-84a6-4a10-9ada-96b69e362f7f None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:40.902649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be6b29db-e1db-4a9e-bfb0-d5ea6a1aa004 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:40.903319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6b29db-e1db-4a9e-bfb0-d5ea6a1aa004 None admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:40.903602 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6b29db-e1db-4a9e-bfb0-d5ea6a1aa004 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:40.903880 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6b29db-e1db-4a9e-bfb0-d5ea6a1aa004 None admin] PATH_INFO: `/v3/projects/455d1c826b974bc38497d18a516867c7/users/688ceb201d304c2da5373c3cf36c2d79/roles/4e19a68eecb04cb998e6aa7adfb48ad3` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:40.906071 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be6b29db-e1db-4a9e-bfb0-d5ea6a1aa004 None admin] RBAC: Authorizing `identity:create_grant(project_id=455d1c826b974bc38497d18a516867c7, user_id=688ceb201d304c2da5373c3cf36c2d79, role_id=4e19a68eecb04cb998e6aa7adfb48ad3)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:40.907550 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be6b29db-e1db-4a9e-bfb0-d5ea6a1aa004 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:40.930426 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 527/1074] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Oct 28 09:32:40 2022] PUT /identity/v3/projects/455d1c826b974bc38497d18a516867c7/users/688ceb201d304c2da5373c3cf36c2d79/roles/4e19a68eecb04cb998e6aa7adfb48ad3 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:32:40.931880 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:40.952484 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f6e4c3-a3d9-4272-8121-81836d017703 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:40.992978 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5949ec-dcb7-45f0-88d2-84cf4c585757 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:40.993717 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5949ec-dcb7-45f0-88d2-84cf4c585757 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:40.993918 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5949ec-dcb7-45f0-88d2-84cf4c585757 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:40.994179 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5949ec-dcb7-45f0-88d2-84cf4c585757 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:41.007805 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b5949ec-dcb7-45f0-88d2-84cf4c585757 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:32:41.012934 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-6b5949ec-dcb7-45f0-88d2-84cf4c585757 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:32:41.012934 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:41.012934 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:41.012934 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:41.012934 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:41.012934 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:41.012934 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:41.012934 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:41.012934 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:41.012934 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:41.012934 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:41.012934 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:41.012934 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:32:41.012934 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:32:41.012934 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:32:41.012934 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:32:41.013954 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:32:41.013954 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:32:41.013954 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:32:41.013954 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:32:41.013954 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:41.013954 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:41.013954 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:41.013954 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:41.013954 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:41.013954 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:41.013954 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:41.013954 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:41.013954 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:41.013954 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:41.013954 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:41.013954 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:41.019472 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:41.019472 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:41.019472 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:41.019472 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:41.019472 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:32:41.019472 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:32:41.019472 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:32:41.019472 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:32:41.019472 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:32:41.019472 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:32:41.019472 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:32:41.019472 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:41.019472 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 548/1075] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Oct 28 09:32:40 2022] GET /identity/v3/roles/admin => generated 83 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:32:41.021333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be6b29db-e1db-4a9e-bfb0-d5ea6a1aa004 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:41.138711 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be6b29db-e1db-4a9e-bfb0-d5ea6a1aa004 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:41.180849 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce8492a2-daea-4526-865d-ef9d4e197778 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:41.181481 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8492a2-daea-4526-865d-ef9d4e197778 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:41.181765 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8492a2-daea-4526-865d-ef9d4e197778 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:41.182041 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8492a2-daea-4526-865d-ef9d4e197778 None admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:41.182753 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce8492a2-daea-4526-865d-ef9d4e197778 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:41.183064 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce8492a2-daea-4526-865d-ef9d4e197778 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:41.184459 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce8492a2-daea-4526-865d-ef9d4e197778 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:41.194994 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 528/1076] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:32:41 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:41.201391 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5949ec-dcb7-45f0-88d2-84cf4c585757 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:41.262171 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1623dc9-5c15-460e-872b-71fde08f7f93 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:41.262641 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1623dc9-5c15-460e-872b-71fde08f7f93 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:41.262927 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1623dc9-5c15-460e-872b-71fde08f7f93 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:41.264421 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 529/1077] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:32:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 09:32:41.270086 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e7506c-8983-4485-a97e-046e7ba87e01 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:41.270372 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e7506c-8983-4485-a97e-046e7ba87e01 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:41.270682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e7506c-8983-4485-a97e-046e7ba87e01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:41.279453 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:41.294961 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-e0e7506c-8983-4485-a97e-046e7ba87e01 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:32:41.296275 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-e0e7506c-8983-4485-a97e-046e7ba87e01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:32:41.312966 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5949ec-dcb7-45f0-88d2-84cf4c585757 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:41.353360 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90929fcd-6cca-43b6-9c89-f663c21c33fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:41.354033 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90929fcd-6cca-43b6-9c89-f663c21c33fc None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:41.354316 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90929fcd-6cca-43b6-9c89-f663c21c33fc None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:41.354623 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90929fcd-6cca-43b6-9c89-f663c21c33fc None admin] PATH_INFO: `/v3/users/253ac57d64d9408c8963a93085b43e7b` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:41.356084 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90929fcd-6cca-43b6-9c89-f663c21c33fc None admin] RBAC: Authorizing `identity:get_user(user_id=253ac57d64d9408c8963a93085b43e7b)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:41.357705 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90929fcd-6cca-43b6-9c89-f663c21c33fc None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:41.364383 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 549/1078] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:32:41 2022] GET /identity/v3/users/253ac57d64d9408c8963a93085b43e7b => generated 285 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:41.365291 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90929fcd-6cca-43b6-9c89-f663c21c33fc None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:41.375287 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:41.382452 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 530/1079] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:32:41 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 09:32:41.486783 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90929fcd-6cca-43b6-9c89-f663c21c33fc None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:41.522994 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844a29c1-de3f-4b96-9798-2378dea28464 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:41.523798 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844a29c1-de3f-4b96-9798-2378dea28464 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:41.524185 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844a29c1-de3f-4b96-9798-2378dea28464 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:41.524564 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-844a29c1-de3f-4b96-9798-2378dea28464 None admin] PATH_INFO: `/v3/projects/506899eec53d4333a54f3567f37027a4` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:41.525879 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-844a29c1-de3f-4b96-9798-2378dea28464 None admin] RBAC: Authorizing `identity:get_project(project_id=506899eec53d4333a54f3567f37027a4)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:41.527445 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-844a29c1-de3f-4b96-9798-2378dea28464 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:41.529381 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 550/1080] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:32:41 2022] GET /identity/v3/projects/506899eec53d4333a54f3567f37027a4 => generated 311 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:32:41.535445 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e7506c-8983-4485-a97e-046e7ba87e01 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:41.648624 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e7506c-8983-4485-a97e-046e7ba87e01 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:41.666068 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844a29c1-de3f-4b96-9798-2378dea28464 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:41.687808 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdfd4559-81e9-4b01-bd91-a49e61898b67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:41.688484 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfd4559-81e9-4b01-bd91-a49e61898b67 None admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:41.688769 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfd4559-81e9-4b01-bd91-a49e61898b67 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:41.689048 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfd4559-81e9-4b01-bd91-a49e61898b67 None admin] PATH_INFO: `/v3/projects/506899eec53d4333a54f3567f37027a4/users/253ac57d64d9408c8963a93085b43e7b/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:41.691013 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdfd4559-81e9-4b01-bd91-a49e61898b67 None admin] RBAC: Authorizing `identity:create_grant(project_id=506899eec53d4333a54f3567f37027a4, user_id=253ac57d64d9408c8963a93085b43e7b, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:41.692474 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdfd4559-81e9-4b01-bd91-a49e61898b67 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:41.695594 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:41.709003 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 531/1081] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Oct 28 09:32:41 2022] PUT /identity/v3/projects/506899eec53d4333a54f3567f37027a4/users/253ac57d64d9408c8963a93085b43e7b/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:32:41.758734 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:41.791061 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-844a29c1-de3f-4b96-9798-2378dea28464 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:41.829853 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7013e9-8c62-49d1-9452-8a87d254d7f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:41.830533 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7013e9-8c62-49d1-9452-8a87d254d7f9 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:41.830914 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7013e9-8c62-49d1-9452-8a87d254d7f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:41.831104 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7013e9-8c62-49d1-9452-8a87d254d7f9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:41.913643 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c7013e9-8c62-49d1-9452-8a87d254d7f9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:41.915714 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c7013e9-8c62-49d1-9452-8a87d254d7f9 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:41.996382 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:42.026531 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 551/1082] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:32:41 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 359 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 28 09:32:42.031243 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdfd4559-81e9-4b01-bd91-a49e61898b67 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:42.147857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdfd4559-81e9-4b01-bd91-a49e61898b67 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:42.183206 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eefdaad3-c9ba-4686-89e2-468d8983ca1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:42.183866 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefdaad3-c9ba-4686-89e2-468d8983ca1c None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:42.184151 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefdaad3-c9ba-4686-89e2-468d8983ca1c None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:42.184425 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefdaad3-c9ba-4686-89e2-468d8983ca1c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:42.196788 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eefdaad3-c9ba-4686-89e2-468d8983ca1c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:42.198326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eefdaad3-c9ba-4686-89e2-468d8983ca1c None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:42.215849 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-eefdaad3-c9ba-4686-89e2-468d8983ca1c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:32:42.215849 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:42.215849 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:42.215849 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:42.215849 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:42.215849 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:42.215849 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:42.215849 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:42.215849 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:42.215849 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:42.215849 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:42.215849 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:42.215849 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:32:42.215849 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:32:42.215849 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:32:42.215849 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:32:42.216902 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:42.216902 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:42.216902 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:42.216902 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:42.216902 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:42.216902 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:42.216902 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:42.216902 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:42.216902 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:42.216902 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:42.216902 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:42.216902 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:42.216902 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:42.216902 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:42.216902 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:42.216902 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:42.216902 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:32:42.222795 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:32:42.222795 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:32:42.222795 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:42.222795 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 532/1083] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:32:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:42.225033 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7013e9-8c62-49d1-9452-8a87d254d7f9 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:42.332498 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7013e9-8c62-49d1-9452-8a87d254d7f9 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:42.370413 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e88adb-db90-45e4-b3ad-86b9b3461a8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:42.371060 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e88adb-db90-45e4-b3ad-86b9b3461a8f None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:42.371343 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e88adb-db90-45e4-b3ad-86b9b3461a8f None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:42.371616 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e88adb-db90-45e4-b3ad-86b9b3461a8f None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:42.372274 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98e88adb-db90-45e4-b3ad-86b9b3461a8f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:42.372583 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98e88adb-db90-45e4-b3ad-86b9b3461a8f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:42.374057 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98e88adb-db90-45e4-b3ad-86b9b3461a8f None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:42.394453 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 552/1084] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:32:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:32:43.144391 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d075c24-f762-47c9-95cb-e70bba17ab67 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:43.144921 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d075c24-f762-47c9-95cb-e70bba17ab67 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:43.144921 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d075c24-f762-47c9-95cb-e70bba17ab67 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:43.146248 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 533/1085] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:32:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:32:43.151782 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94944ee-d36f-40d9-b3b7-af0499004c01 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:43.152224 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94944ee-d36f-40d9-b3b7-af0499004c01 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:43.152318 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94944ee-d36f-40d9-b3b7-af0499004c01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:43.172178 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-a94944ee-d36f-40d9-b3b7-af0499004c01 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:32:43.173363 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a94944ee-d36f-40d9-b3b7-af0499004c01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:32:43.199517 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:43.239178 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 553/1086] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:32:43 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:32:43.510747 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d075c24-f762-47c9-95cb-e70bba17ab67 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:43.561648 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:43.617834 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d075c24-f762-47c9-95cb-e70bba17ab67 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:43.663955 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d895be5-f2e6-441e-a991-1d6dd62bf47a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:43.664594 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d895be5-f2e6-441e-a991-1d6dd62bf47a None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:43.664874 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d895be5-f2e6-441e-a991-1d6dd62bf47a None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:43.665146 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d895be5-f2e6-441e-a991-1d6dd62bf47a None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:43.675677 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d895be5-f2e6-441e-a991-1d6dd62bf47a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:32:43.680983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-5d895be5-f2e6-441e-a991-1d6dd62bf47a None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 28 09:32:43.680983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:43.680983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:43.680983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:43.680983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:43.680983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:43.680983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:43.680983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:43.680983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:43.680983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:43.680983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:43.680983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:43.680983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:32:43.680983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:32:43.680983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:32:43.680983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:32:43.681994 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:32:43.681994 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:32:43.681994 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:32:43.681994 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:32:43.681994 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:43.681994 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:43.681994 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:43.681994 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:43.681994 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:43.681994 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:43.681994 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:43.681994 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:43.681994 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:43.681994 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:43.681994 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:43.681994 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:43.682889 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:43.682889 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:43.682889 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:43.682889 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:43.682889 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:32:43.682889 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:32:43.682889 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:32:43.682889 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:32:43.682889 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:32:43.682889 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:32:43.682889 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Oct 28 09:32:43.682889 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:43.682889 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 534/1087] 10.222.0.25 () {56 vars in 1120 bytes} [Fri Oct 28 09:32:43 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:43.688240 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94944ee-d36f-40d9-b3b7-af0499004c01 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:43.786587 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94944ee-d36f-40d9-b3b7-af0499004c01 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:43.824595 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84807caf-2287-4e43-83ca-c87ffe5351d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:43.825226 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84807caf-2287-4e43-83ca-c87ffe5351d6 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:43.825503 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84807caf-2287-4e43-83ca-c87ffe5351d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:43.825771 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84807caf-2287-4e43-83ca-c87ffe5351d6 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:43.826466 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84807caf-2287-4e43-83ca-c87ffe5351d6 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:43.826770 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84807caf-2287-4e43-83ca-c87ffe5351d6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:43.828145 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84807caf-2287-4e43-83ca-c87ffe5351d6 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:43.837048 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 554/1088] 10.222.0.25 () {56 vars in 1115 bytes} [Fri Oct 28 09:32:43 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:43.842978 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d895be5-f2e6-441e-a991-1d6dd62bf47a None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:43.922506 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9eac0e-871c-4ed4-b88c-a77f0a90e9da None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:43.922793 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9eac0e-871c-4ed4-b88c-a77f0a90e9da None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:43.923072 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9eac0e-871c-4ed4-b88c-a77f0a90e9da None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:43.924421 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 555/1089] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:32:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 09:32:43.930304 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a820aeb-15ee-452f-9197-16581e1a0fee None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:43.930608 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a820aeb-15ee-452f-9197-16581e1a0fee None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:43.930906 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a820aeb-15ee-452f-9197-16581e1a0fee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:43.949344 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-2a820aeb-15ee-452f-9197-16581e1a0fee None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:32:43.950497 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d895be5-f2e6-441e-a991-1d6dd62bf47a None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:43.950626 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-2a820aeb-15ee-452f-9197-16581e1a0fee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:32:43.958260 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:43.972072 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:43.987499 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f8ae89-d2d0-40fd-bdd7-1681b5f81767 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:43.988147 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f8ae89-d2d0-40fd-bdd7-1681b5f81767 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:43.988431 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f8ae89-d2d0-40fd-bdd7-1681b5f81767 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:43.988719 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f8ae89-d2d0-40fd-bdd7-1681b5f81767 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:43.999339 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59f8ae89-d2d0-40fd-bdd7-1681b5f81767 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:44.000837 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59f8ae89-d2d0-40fd-bdd7-1681b5f81767 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:44.016623 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-59f8ae89-d2d0-40fd-bdd7-1681b5f81767 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:32:44.016623 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:44.016623 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:44.016623 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:44.016623 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:44.016623 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:44.016623 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:44.016623 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:44.016623 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:44.016623 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:44.016623 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:44.016623 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:44.016623 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:32:44.016623 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:32:44.016623 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:32:44.016623 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:32:44.017628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:44.017628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:44.017628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:44.017628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:44.017628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:44.017628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:44.017628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:44.017628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:44.017628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:44.017628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:44.017628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:44.017628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:44.017628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:44.017628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:44.017628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:44.017628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:44.017628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:32:44.018584 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:32:44.018584 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:32:44.018584 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:44.018584 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 535/1090] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:32:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:44.020723 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 556/1091] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:32:43 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 09:32:44.028863 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f8ae89-d2d0-40fd-bdd7-1681b5f81767 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:44.129241 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f8ae89-d2d0-40fd-bdd7-1681b5f81767 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:44.164972 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329af800-1d22-42dc-9f46-eca6336d01fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:44.165617 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329af800-1d22-42dc-9f46-eca6336d01fe None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:44.165849 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329af800-1d22-42dc-9f46-eca6336d01fe None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:44.166125 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329af800-1d22-42dc-9f46-eca6336d01fe None admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:44.166804 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-329af800-1d22-42dc-9f46-eca6336d01fe None admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:44.167110 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-329af800-1d22-42dc-9f46-eca6336d01fe None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:44.168482 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-329af800-1d22-42dc-9f46-eca6336d01fe None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:44.180794 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 536/1092] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:32:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:44.186719 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a820aeb-15ee-452f-9197-16581e1a0fee None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:44.283171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a820aeb-15ee-452f-9197-16581e1a0fee None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:44.294934 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329af800-1d22-42dc-9f46-eca6336d01fe None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:44.315708 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:44.318017 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad4f10d-9f24-4c4b-940f-7768ebb8108b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:44.319256 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad4f10d-9f24-4c4b-940f-7768ebb8108b None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:44.319545 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad4f10d-9f24-4c4b-940f-7768ebb8108b None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:44.319821 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad4f10d-9f24-4c4b-940f-7768ebb8108b None admin] PATH_INFO: `/v3/users/nova` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:44.327743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ad4f10d-9f24-4c4b-940f-7768ebb8108b None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:44.329173 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ad4f10d-9f24-4c4b-940f-7768ebb8108b None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:44.341113 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-1ad4f10d-9f24-4c4b-940f-7768ebb8108b None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Oct 28 09:32:44.341113 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:44.341113 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:44.341113 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:44.341113 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:44.341113 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:44.341113 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:44.341113 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:44.341113 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:44.341113 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:44.341113 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:44.341113 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:44.341113 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 09:32:44.341113 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 09:32:44.341113 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 09:32:44.341113 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 09:32:44.342281 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:44.342281 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:44.342281 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 09:32:44.342281 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:32:44.342281 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 09:32:44.342281 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:32:44.342281 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:44.342281 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:44.342281 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:44.342281 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:44.342281 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:44.342281 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:44.342281 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:44.342281 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:44.342281 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:44.342281 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:44.342281 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:44.342281 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:44.343273 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:44.343273 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:44.343273 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 09:32:44.343273 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 09:32:44.343273 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 09:32:44.343273 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 09:32:44.343273 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 09:32:44.343273 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 09:32:44.343273 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Oct 28 09:32:44.343273 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:44.343273 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 557/1093] 10.222.0.25 () {56 vars in 1093 bytes} [Fri Oct 28 09:32:44 2022] GET /identity/v3/users/nova => generated 82 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:32:44.348109 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad4f10d-9f24-4c4b-940f-7768ebb8108b None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:44.367764 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:44.403798 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329af800-1d22-42dc-9f46-eca6336d01fe None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:44.441328 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9ab887-b89e-415e-82f7-bf34239d1786 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:44.441976 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9ab887-b89e-415e-82f7-bf34239d1786 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:44.442253 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9ab887-b89e-415e-82f7-bf34239d1786 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:44.442561 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9ab887-b89e-415e-82f7-bf34239d1786 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:44.452219 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad4f10d-9f24-4c4b-940f-7768ebb8108b None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:44.452820 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef9ab887-b89e-415e-82f7-bf34239d1786 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:32:44.457633 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-ef9ab887-b89e-415e-82f7-bf34239d1786 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:32:44.457633 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:44.457633 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:44.457633 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:44.457633 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:44.457633 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:44.457633 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:44.457633 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:44.457633 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:44.457633 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:44.457633 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:44.457633 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:44.457633 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:32:44.457633 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:32:44.457633 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:32:44.457633 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:32:44.458892 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:32:44.458892 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:32:44.458892 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:32:44.458892 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:32:44.458892 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:44.458892 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:44.458892 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:44.458892 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:44.458892 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:44.458892 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:44.458892 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:44.458892 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:44.458892 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:44.458892 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:44.458892 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:44.458892 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:44.459974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:44.459974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:44.459974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:44.459974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:44.459974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:32:44.459974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:32:44.459974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:32:44.459974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:32:44.459974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:32:44.459974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:32:44.459974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:32:44.459974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:32:44.459974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 537/1094] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Oct 28 09:32:44 2022] GET /identity/v3/roles/admin => generated 83 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:32:44.464888 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9ab887-b89e-415e-82f7-bf34239d1786 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:44.489681 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2253699-020f-4e55-a07b-b0af404c1a66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:44.490893 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2253699-020f-4e55-a07b-b0af404c1a66 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:44.491178 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2253699-020f-4e55-a07b-b0af404c1a66 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:44.491454 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2253699-020f-4e55-a07b-b0af404c1a66 None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:44.492201 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2253699-020f-4e55-a07b-b0af404c1a66 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:44.492510 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2253699-020f-4e55-a07b-b0af404c1a66 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:44.493967 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2253699-020f-4e55-a07b-b0af404c1a66 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:44.512607 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 558/1095] 10.222.0.25 () {56 vars in 1133 bytes} [Fri Oct 28 09:32:44 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:32:44.518585 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2253699-020f-4e55-a07b-b0af404c1a66 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:44.547024 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:44.569966 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9ab887-b89e-415e-82f7-bf34239d1786 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:44.600819 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:44.606825 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eab86fe-7c63-4990-bb99-bc2b11b2fc4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:44.607444 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eab86fe-7c63-4990-bb99-bc2b11b2fc4f None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:44.607725 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eab86fe-7c63-4990-bb99-bc2b11b2fc4f None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:44.608296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eab86fe-7c63-4990-bb99-bc2b11b2fc4f None admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:44.608688 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eab86fe-7c63-4990-bb99-bc2b11b2fc4f None admin] RBAC: Adding query filter params (name=admin) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:44.609001 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eab86fe-7c63-4990-bb99-bc2b11b2fc4f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:44.610415 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eab86fe-7c63-4990-bb99-bc2b11b2fc4f None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:44.618887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 538/1096] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:32:44 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:32:44.624809 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eab86fe-7c63-4990-bb99-bc2b11b2fc4f None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:44.626334 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2253699-020f-4e55-a07b-b0af404c1a66 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:44.662070 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7e6b72-ffc7-4ae8-ac52-d04540a318e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:44.662613 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7e6b72-ffc7-4ae8-ac52-d04540a318e3 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:44.662893 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7e6b72-ffc7-4ae8-ac52-d04540a318e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:44.663161 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7e6b72-ffc7-4ae8-ac52-d04540a318e3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:44.726765 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eab86fe-7c63-4990-bb99-bc2b11b2fc4f None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:44.733415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b7e6b72-ffc7-4ae8-ac52-d04540a318e3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:44.734888 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b7e6b72-ffc7-4ae8-ac52-d04540a318e3 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:44.761196 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f5244e1-2999-4acf-a92a-43e12bdf4348 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:44.761834 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5244e1-2999-4acf-a92a-43e12bdf4348 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:44.762115 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5244e1-2999-4acf-a92a-43e12bdf4348 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:44.762402 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5244e1-2999-4acf-a92a-43e12bdf4348 None admin] PATH_INFO: `/v3/users/253ac57d64d9408c8963a93085b43e7b` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:44.763555 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f5244e1-2999-4acf-a92a-43e12bdf4348 None admin] RBAC: Authorizing `identity:get_user(user_id=253ac57d64d9408c8963a93085b43e7b)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:44.764996 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f5244e1-2999-4acf-a92a-43e12bdf4348 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:44.766919 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 539/1097] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:32:44 2022] GET /identity/v3/users/253ac57d64d9408c8963a93085b43e7b => generated 285 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:32:44.772637 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f5244e1-2999-4acf-a92a-43e12bdf4348 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:44.783804 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:44.809511 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:44.836618 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 559/1098] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:32:44 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 28 09:32:44.842420 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7e6b72-ffc7-4ae8-ac52-d04540a318e3 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:44.875205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f5244e1-2999-4acf-a92a-43e12bdf4348 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:44.909030 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80dea89c-2d7d-4bdd-af25-916723f8b8b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:44.909640 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80dea89c-2d7d-4bdd-af25-916723f8b8b8 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:44.909913 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80dea89c-2d7d-4bdd-af25-916723f8b8b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:44.910186 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80dea89c-2d7d-4bdd-af25-916723f8b8b8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:44.941785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7e6b72-ffc7-4ae8-ac52-d04540a318e3 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:44.976004 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad01621f-8220-409b-ace2-27e25f347281 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:44.976653 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad01621f-8220-409b-ace2-27e25f347281 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:44.976940 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad01621f-8220-409b-ace2-27e25f347281 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:44.977209 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad01621f-8220-409b-ace2-27e25f347281 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:44.977898 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80dea89c-2d7d-4bdd-af25-916723f8b8b8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:44.979455 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80dea89c-2d7d-4bdd-af25-916723f8b8b8 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:44.989379 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad01621f-8220-409b-ace2-27e25f347281 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:44.990438 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:44.990930 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad01621f-8220-409b-ace2-27e25f347281 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:45.008073 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-ad01621f-8220-409b-ace2-27e25f347281 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:32:45.008073 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:45.008073 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:45.008073 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:45.008073 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:45.008073 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:45.008073 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:45.008073 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:45.008073 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:45.008073 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:45.008073 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:45.008073 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:45.008073 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:32:45.008073 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:32:45.008073 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:32:45.008073 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:32:45.009134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:45.009134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:45.009134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:45.009134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:45.009134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:45.009134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:45.009134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:45.009134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:45.009134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:45.009134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:45.009134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:45.009134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:45.009134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:45.009134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:45.009134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:45.009134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:45.009134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:32:45.010023 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:32:45.010023 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:32:45.010023 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:45.010023 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 560/1099] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:32:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:32:45.014950 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad01621f-8220-409b-ace2-27e25f347281 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:45.043817 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:45.086841 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 540/1100] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:32:44 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Oct 28 09:32:45.093328 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80dea89c-2d7d-4bdd-af25-916723f8b8b8 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:45.120265 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad01621f-8220-409b-ace2-27e25f347281 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:45.155531 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e51db7-5453-4998-8554-cd39443fd3ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:45.156139 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e51db7-5453-4998-8554-cd39443fd3ba None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:45.156418 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e51db7-5453-4998-8554-cd39443fd3ba None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:45.156693 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e51db7-5453-4998-8554-cd39443fd3ba None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:45.157328 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e51db7-5453-4998-8554-cd39443fd3ba None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:45.157633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e51db7-5453-4998-8554-cd39443fd3ba None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:45.159268 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e51db7-5453-4998-8554-cd39443fd3ba None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:45.171626 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 561/1101] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:32:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:32:45.177323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e51db7-5453-4998-8554-cd39443fd3ba None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:45.199252 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80dea89c-2d7d-4bdd-af25-916723f8b8b8 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:45.208570 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:45.235018 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e63b9c6-5e98-4344-b319-38c201221baf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:45.235653 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e63b9c6-5e98-4344-b319-38c201221baf None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:45.235931 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e63b9c6-5e98-4344-b319-38c201221baf None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:45.236201 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e63b9c6-5e98-4344-b319-38c201221baf None admin] PATH_INFO: `/v3/projects/506899eec53d4333a54f3567f37027a4` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:45.237446 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e63b9c6-5e98-4344-b319-38c201221baf None admin] RBAC: Authorizing `identity:get_project(project_id=506899eec53d4333a54f3567f37027a4)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:45.238928 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e63b9c6-5e98-4344-b319-38c201221baf None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:45.241386 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 541/1102] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:32:45 2022] GET /identity/v3/projects/506899eec53d4333a54f3567f37027a4 => generated 311 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:32:45.247397 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e63b9c6-5e98-4344-b319-38c201221baf None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:45.268067 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:45.288336 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e51db7-5453-4998-8554-cd39443fd3ba None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:45.323819 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba6a7619-a5f0-47a5-bccb-4b2140a6def7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:45.324457 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6a7619-a5f0-47a5-bccb-4b2140a6def7 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:45.324731 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6a7619-a5f0-47a5-bccb-4b2140a6def7 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:45.324999 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6a7619-a5f0-47a5-bccb-4b2140a6def7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:45.332000 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba6a7619-a5f0-47a5-bccb-4b2140a6def7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:45.333438 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba6a7619-a5f0-47a5-bccb-4b2140a6def7 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:45.344840 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-ba6a7619-a5f0-47a5-bccb-4b2140a6def7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:32:45.344840 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:32:45.344840 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:32:45.344840 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:32:45.344840 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:32:45.344840 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:32:45.344840 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:32:45.344840 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:32:45.344840 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:32:45.344840 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:32:45.344840 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:32:45.344840 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:32:45.344840 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 09:32:45.344840 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 09:32:45.344840 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 09:32:45.345785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 09:32:45.345785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:32:45.345785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:32:45.345785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:32:45.345785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:32:45.345785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:32:45.345785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:32:45.345785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:32:45.345785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:32:45.345785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:32:45.345785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:32:45.345785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:32:45.345785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:32:45.345785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:32:45.345785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:32:45.345785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:32:45.345785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:32:45.346805 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 09:32:45.346805 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 09:32:45.346805 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 09:32:45.346805 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 09:32:45.346805 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 09:32:45.346805 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 09:32:45.346805 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:32:45.346805 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:32:45.346805 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 562/1103] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:32:45 2022] GET /identity/v3/projects/service => generated 88 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:32:45.352289 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba6a7619-a5f0-47a5-bccb-4b2140a6def7 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:45.357622 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e63b9c6-5e98-4344-b319-38c201221baf None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:45.393807 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec1e20e-c266-4dc7-83b0-4f681bfa17fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:45.394436 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec1e20e-c266-4dc7-83b0-4f681bfa17fd None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:45.394723 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec1e20e-c266-4dc7-83b0-4f681bfa17fd None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:45.395003 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec1e20e-c266-4dc7-83b0-4f681bfa17fd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:45.395730 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec1e20e-c266-4dc7-83b0-4f681bfa17fd None admin] RBAC: Adding query filter params (role.id=e13d798a98ea4c3dbb87a236ec683d32, scope.project.id=506899eec53d4333a54f3567f37027a4, user.id=253ac57d64d9408c8963a93085b43e7b) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:45.396038 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec1e20e-c266-4dc7-83b0-4f681bfa17fd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:45.397480 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec1e20e-c266-4dc7-83b0-4f681bfa17fd None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:45.403400 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 542/1104] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:32:45 2022] GET /identity/v3/role_assignments?user.id=253ac57d64d9408c8963a93085b43e7b&scope.project.id=506899eec53d4333a54f3567f37027a4&role.id=e13d798a98ea4c3dbb87a236ec683d32 => generated 608 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:32:45.456847 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba6a7619-a5f0-47a5-bccb-4b2140a6def7 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:45.495191 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acb29cd1-4967-44fe-b893-686e37461139 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:45.495808 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb29cd1-4967-44fe-b893-686e37461139 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:45.496087 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb29cd1-4967-44fe-b893-686e37461139 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:45.496362 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acb29cd1-4967-44fe-b893-686e37461139 None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:45.497065 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb29cd1-4967-44fe-b893-686e37461139 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:45.497364 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb29cd1-4967-44fe-b893-686e37461139 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:45.498843 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acb29cd1-4967-44fe-b893-686e37461139 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:45.512318 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 563/1105] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Oct 28 09:32:45 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:32:45.518894 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec1e20e-c266-4dc7-83b0-4f681bfa17fd None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:45.568675 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:45.631046 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec1e20e-c266-4dc7-83b0-4f681bfa17fd None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:45.666740 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2644476b-e79d-44c6-b098-ba73bd6c270f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:45.667352 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2644476b-e79d-44c6-b098-ba73bd6c270f None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:45.667633 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2644476b-e79d-44c6-b098-ba73bd6c270f None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:45.667907 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2644476b-e79d-44c6-b098-ba73bd6c270f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:45.668633 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2644476b-e79d-44c6-b098-ba73bd6c270f None admin] RBAC: Adding query filter params (role.id=4e19a68eecb04cb998e6aa7adfb48ad3, scope.project.id=455d1c826b974bc38497d18a516867c7, user.id=688ceb201d304c2da5373c3cf36c2d79) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:45.668941 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2644476b-e79d-44c6-b098-ba73bd6c270f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:45.670371 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2644476b-e79d-44c6-b098-ba73bd6c270f None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:45.676290 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 543/1106] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:32:45 2022] GET /identity/v3/role_assignments?user.id=688ceb201d304c2da5373c3cf36c2d79&scope.project.id=455d1c826b974bc38497d18a516867c7&role.id=4e19a68eecb04cb998e6aa7adfb48ad3 => generated 608 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:47.623251 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd89ec1-fb3f-4810-9bff-09fd7782cc92 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:47.623784 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd89ec1-fb3f-4810-9bff-09fd7782cc92 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:47.623784 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd89ec1-fb3f-4810-9bff-09fd7782cc92 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:47.625132 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 564/1107] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:32:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:32:47.630359 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f991d87-9407-4dde-99a1-6d51590e0cd9 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:47.631302 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f991d87-9407-4dde-99a1-6d51590e0cd9 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:47.631462 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f991d87-9407-4dde-99a1-6d51590e0cd9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:47.649650 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-4f991d87-9407-4dde-99a1-6d51590e0cd9 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:32:47.650884 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-4f991d87-9407-4dde-99a1-6d51590e0cd9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:32:47.716953 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 544/1108] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:32:47 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:32:47.990921 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd89ec1-fb3f-4810-9bff-09fd7782cc92 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:47.999875 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:48.093380 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd89ec1-fb3f-4810-9bff-09fd7782cc92 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:48.128038 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31611e82-ffbd-4bf0-a2c6-71eb615dcd2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:48.128754 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31611e82-ffbd-4bf0-a2c6-71eb615dcd2d None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:48.129036 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31611e82-ffbd-4bf0-a2c6-71eb615dcd2d None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:48.129309 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31611e82-ffbd-4bf0-a2c6-71eb615dcd2d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:48.130483 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31611e82-ffbd-4bf0-a2c6-71eb615dcd2d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:48.132422 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31611e82-ffbd-4bf0-a2c6-71eb615dcd2d None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:48.134002 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 565/1109] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:32:47 2022] GET /identity/v3/domains/default => generated 199 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:48.140075 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f991d87-9407-4dde-99a1-6d51590e0cd9 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:48.174063 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:48.243607 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f991d87-9407-4dde-99a1-6d51590e0cd9 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:48.277504 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97c4e88-d749-4f80-baeb-b50565dac7fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:48.278113 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97c4e88-d749-4f80-baeb-b50565dac7fc None admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:48.278388 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97c4e88-d749-4f80-baeb-b50565dac7fc None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:48.278703 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97c4e88-d749-4f80-baeb-b50565dac7fc None admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:48.279381 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b97c4e88-d749-4f80-baeb-b50565dac7fc None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:48.280798 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b97c4e88-d749-4f80-baeb-b50565dac7fc None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:48.302846 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 545/1110] 10.222.0.25 () {60 vars in 1132 bytes} [Fri Oct 28 09:32:48 2022] POST /identity/v3/users => generated 286 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:32:50.496031 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251e84d5-448f-4f56-9d26-bb7685efdf46 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:50.496634 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251e84d5-448f-4f56-9d26-bb7685efdf46 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:50.496634 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-251e84d5-448f-4f56-9d26-bb7685efdf46 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:50.497907 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 566/1111] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:32:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:32:50.503286 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e164bd0-c830-4c27-bed4-6480e606f7c5 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:50.503675 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e164bd0-c830-4c27-bed4-6480e606f7c5 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:50.503838 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e164bd0-c830-4c27-bed4-6480e606f7c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:50.523081 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-7e164bd0-c830-4c27-bed4-6480e606f7c5 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:32:50.524310 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-7e164bd0-c830-4c27-bed4-6480e606f7c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:32:50.592710 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 546/1112] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:32:50 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:32:50.860772 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251e84d5-448f-4f56-9d26-bb7685efdf46 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:50.934012 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:50.960448 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-251e84d5-448f-4f56-9d26-bb7685efdf46 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:50.993794 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668f4e30-246a-4c29-bda4-6ebbeed75a5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:50.994460 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668f4e30-246a-4c29-bda4-6ebbeed75a5a None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:50.994748 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668f4e30-246a-4c29-bda4-6ebbeed75a5a None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:50.995028 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-668f4e30-246a-4c29-bda4-6ebbeed75a5a None admin] PATH_INFO: `/v3/roles/d93b4c94b590457199c3d301f947b0fb` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:50.996415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-668f4e30-246a-4c29-bda4-6ebbeed75a5a None admin] RBAC: Authorizing `identity:get_role(role_id=d93b4c94b590457199c3d301f947b0fb)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:50.997756 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-668f4e30-246a-4c29-bda4-6ebbeed75a5a None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:50.999108 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 567/1113] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:32:50 2022] GET /identity/v3/roles/d93b4c94b590457199c3d301f947b0fb => generated 223 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:51.004913 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e164bd0-c830-4c27-bed4-6480e606f7c5 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:51.025717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:51.105372 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e164bd0-c830-4c27-bed4-6480e606f7c5 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:51.138596 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a6c191f-4eb2-4ba8-8457-6f74212642a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:51.139154 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6c191f-4eb2-4ba8-8457-6f74212642a6 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:51.139435 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6c191f-4eb2-4ba8-8457-6f74212642a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:51.139708 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6c191f-4eb2-4ba8-8457-6f74212642a6 None admin] PATH_INFO: `/v3/users/1f4e105fbd3c43d4b25843da163b3dd3` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:51.152881 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a6c191f-4eb2-4ba8-8457-6f74212642a6 None admin] RBAC: Authorizing `identity:get_user(user_id=1f4e105fbd3c43d4b25843da163b3dd3)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:51.154320 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a6c191f-4eb2-4ba8-8457-6f74212642a6 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:51.155946 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 547/1114] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:32:51 2022] GET /identity/v3/users/1f4e105fbd3c43d4b25843da163b3dd3 => generated 286 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:51.162037 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668f4e30-246a-4c29-bda4-6ebbeed75a5a None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:51.258076 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-668f4e30-246a-4c29-bda4-6ebbeed75a5a None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:51.290282 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f26c2ef-ba08-4e81-ab7d-50b1c4d5bb9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:51.290898 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f26c2ef-ba08-4e81-ab7d-50b1c4d5bb9e None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:51.291175 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f26c2ef-ba08-4e81-ab7d-50b1c4d5bb9e None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:51.291446 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f26c2ef-ba08-4e81-ab7d-50b1c4d5bb9e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:51.302757 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:51.356331 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f26c2ef-ba08-4e81-ab7d-50b1c4d5bb9e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:51.357756 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f26c2ef-ba08-4e81-ab7d-50b1c4d5bb9e None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:51.451620 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 568/1115] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:32:51 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 290 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 09:32:51.457523 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a6c191f-4eb2-4ba8-8457-6f74212642a6 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:51.536151 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:51.562891 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a6c191f-4eb2-4ba8-8457-6f74212642a6 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:51.598239 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67165e2-d5c1-4813-a109-2568360ebfc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:51.598899 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67165e2-d5c1-4813-a109-2568360ebfc4 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:51.599178 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67165e2-d5c1-4813-a109-2568360ebfc4 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:51.599515 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67165e2-d5c1-4813-a109-2568360ebfc4 None admin] PATH_INFO: `/v3/projects/506899eec53d4333a54f3567f37027a4` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:51.601111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b67165e2-d5c1-4813-a109-2568360ebfc4 None admin] RBAC: Authorizing `identity:get_project(project_id=506899eec53d4333a54f3567f37027a4)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:51.602584 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b67165e2-d5c1-4813-a109-2568360ebfc4 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:51.604205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 548/1116] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:32:51 2022] GET /identity/v3/projects/506899eec53d4333a54f3567f37027a4 => generated 311 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:51.610370 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f26c2ef-ba08-4e81-ab7d-50b1c4d5bb9e None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:51.651506 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:51.706058 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f26c2ef-ba08-4e81-ab7d-50b1c4d5bb9e None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:51.738510 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf26606d-0354-4178-a656-f2daf566a64d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:51.739106 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf26606d-0354-4178-a656-f2daf566a64d None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:51.739384 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf26606d-0354-4178-a656-f2daf566a64d None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:51.739652 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf26606d-0354-4178-a656-f2daf566a64d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:51.740872 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf26606d-0354-4178-a656-f2daf566a64d None admin] RBAC: Adding query filter params (role.id=d93b4c94b590457199c3d301f947b0fb, scope.project.id=506899eec53d4333a54f3567f37027a4, user.id=1f4e105fbd3c43d4b25843da163b3dd3) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:51.741173 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf26606d-0354-4178-a656-f2daf566a64d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:51.742586 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf26606d-0354-4178-a656-f2daf566a64d None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:51.747746 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 569/1117] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:32:51 2022] GET /identity/v3/role_assignments?user.id=1f4e105fbd3c43d4b25843da163b3dd3&scope.project.id=506899eec53d4333a54f3567f37027a4&role.id=d93b4c94b590457199c3d301f947b0fb => generated 259 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:53.905579 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c888440-41ca-46a1-81e6-60516353c20c None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:53.906160 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c888440-41ca-46a1-81e6-60516353c20c None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:53.906160 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c888440-41ca-46a1-81e6-60516353c20c None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:53.907497 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 549/1118] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:32:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:32:53.913966 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1150c1-7fda-4c32-8714-768f4e3209c3 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:53.914489 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1150c1-7fda-4c32-8714-768f4e3209c3 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:53.914597 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b1150c1-7fda-4c32-8714-768f4e3209c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:53.933410 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-8b1150c1-7fda-4c32-8714-768f4e3209c3 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:32:53.934659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-8b1150c1-7fda-4c32-8714-768f4e3209c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:32:54.003385 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 570/1119] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:32:53 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:32:54.275789 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c888440-41ca-46a1-81e6-60516353c20c None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:54.378113 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c888440-41ca-46a1-81e6-60516353c20c None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:54.413052 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc88883-ec3e-4a01-a74e-400ee0e9e1a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:54.413678 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc88883-ec3e-4a01-a74e-400ee0e9e1a6 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:54.413960 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc88883-ec3e-4a01-a74e-400ee0e9e1a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:54.414236 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc88883-ec3e-4a01-a74e-400ee0e9e1a6 None admin] PATH_INFO: `/v3/roles/d93b4c94b590457199c3d301f947b0fb` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:54.415655 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc88883-ec3e-4a01-a74e-400ee0e9e1a6 None admin] RBAC: Authorizing `identity:get_role(role_id=d93b4c94b590457199c3d301f947b0fb)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:54.417022 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc88883-ec3e-4a01-a74e-400ee0e9e1a6 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:54.418330 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 550/1120] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:32:54 2022] GET /identity/v3/roles/d93b4c94b590457199c3d301f947b0fb => generated 223 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:54.424277 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b1150c1-7fda-4c32-8714-768f4e3209c3 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:54.458709 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:54.526923 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b1150c1-7fda-4c32-8714-768f4e3209c3 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:54.562065 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-325247f0-a55b-40e4-ad8a-efe263eb5b0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:54.562736 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-325247f0-a55b-40e4-ad8a-efe263eb5b0c None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:54.563020 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-325247f0-a55b-40e4-ad8a-efe263eb5b0c None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:54.563293 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-325247f0-a55b-40e4-ad8a-efe263eb5b0c None admin] PATH_INFO: `/v3/users/1f4e105fbd3c43d4b25843da163b3dd3` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:54.564412 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-325247f0-a55b-40e4-ad8a-efe263eb5b0c None admin] RBAC: Authorizing `identity:get_user(user_id=1f4e105fbd3c43d4b25843da163b3dd3)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:54.565823 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-325247f0-a55b-40e4-ad8a-efe263eb5b0c None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:54.567451 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 571/1121] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:32:54 2022] GET /identity/v3/users/1f4e105fbd3c43d4b25843da163b3dd3 => generated 286 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:54.575847 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc88883-ec3e-4a01-a74e-400ee0e9e1a6 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:54.594317 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:54.681729 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc88883-ec3e-4a01-a74e-400ee0e9e1a6 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:54.715103 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-922bb64e-4e5b-42ba-9993-3e1dd812ee58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:54.715729 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922bb64e-4e5b-42ba-9993-3e1dd812ee58 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:54.716010 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922bb64e-4e5b-42ba-9993-3e1dd812ee58 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:54.716281 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922bb64e-4e5b-42ba-9993-3e1dd812ee58 None admin] PATH_INFO: `/v3/projects/506899eec53d4333a54f3567f37027a4` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:54.717388 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-922bb64e-4e5b-42ba-9993-3e1dd812ee58 None admin] RBAC: Authorizing `identity:get_project(project_id=506899eec53d4333a54f3567f37027a4)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:54.718854 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-922bb64e-4e5b-42ba-9993-3e1dd812ee58 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:54.720462 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 551/1122] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:32:54 2022] GET /identity/v3/projects/506899eec53d4333a54f3567f37027a4 => generated 311 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:54.727866 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-325247f0-a55b-40e4-ad8a-efe263eb5b0c None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:54.829053 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-325247f0-a55b-40e4-ad8a-efe263eb5b0c None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:54.831397 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:54.864329 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84a50ff1-3981-4f36-ab4d-5e921176eb23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:54.865020 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a50ff1-3981-4f36-ab4d-5e921176eb23 None admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:54.865302 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a50ff1-3981-4f36-ab4d-5e921176eb23 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:54.865703 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84a50ff1-3981-4f36-ab4d-5e921176eb23 None admin] PATH_INFO: `/v3/projects/506899eec53d4333a54f3567f37027a4/users/1f4e105fbd3c43d4b25843da163b3dd3/roles/d93b4c94b590457199c3d301f947b0fb` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:54.867918 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84a50ff1-3981-4f36-ab4d-5e921176eb23 None admin] RBAC: Authorizing `identity:create_grant(project_id=506899eec53d4333a54f3567f37027a4, user_id=1f4e105fbd3c43d4b25843da163b3dd3, role_id=d93b4c94b590457199c3d301f947b0fb)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:54.869345 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84a50ff1-3981-4f36-ab4d-5e921176eb23 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:54.884019 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 572/1123] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Oct 28 09:32:54 2022] PUT /identity/v3/projects/506899eec53d4333a54f3567f37027a4/users/1f4e105fbd3c43d4b25843da163b3dd3/roles/d93b4c94b590457199c3d301f947b0fb => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:32:57.044598 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba9c206-05e6-4a21-9681-f864a731878c None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:57.045120 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba9c206-05e6-4a21-9681-f864a731878c None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:57.045120 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba9c206-05e6-4a21-9681-f864a731878c None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:57.046470 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 552/1124] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:32:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:32:57.051911 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4512df3b-ed24-4b44-937c-3b86a8f2dc6e None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:57.052287 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4512df3b-ed24-4b44-937c-3b86a8f2dc6e None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:57.052969 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4512df3b-ed24-4b44-937c-3b86a8f2dc6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:57.071924 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-4512df3b-ed24-4b44-937c-3b86a8f2dc6e None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:32:57.073108 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4512df3b-ed24-4b44-937c-3b86a8f2dc6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:32:57.144371 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 573/1125] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:32:57 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:32:57.417463 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba9c206-05e6-4a21-9681-f864a731878c None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:57.504508 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:57.524505 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba9c206-05e6-4a21-9681-f864a731878c None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:57.558184 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6165410-482f-4123-87ab-578c738980d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:57.558836 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6165410-482f-4123-87ab-578c738980d3 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:57.559208 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6165410-482f-4123-87ab-578c738980d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:57.559373 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6165410-482f-4123-87ab-578c738980d3 None admin] PATH_INFO: `/v3/roles/d93b4c94b590457199c3d301f947b0fb` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:57.560760 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6165410-482f-4123-87ab-578c738980d3 None admin] RBAC: Authorizing `identity:get_role(role_id=d93b4c94b590457199c3d301f947b0fb)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:57.562122 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6165410-482f-4123-87ab-578c738980d3 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:57.563466 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 553/1126] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:32:57 2022] GET /identity/v3/roles/d93b4c94b590457199c3d301f947b0fb => generated 223 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:32:57.569976 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4512df3b-ed24-4b44-937c-3b86a8f2dc6e None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:57.663377 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:57.670476 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4512df3b-ed24-4b44-937c-3b86a8f2dc6e None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:57.704407 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6490b03f-7ec5-4304-922b-6254509b2bc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:57.705028 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6490b03f-7ec5-4304-922b-6254509b2bc0 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:57.705313 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6490b03f-7ec5-4304-922b-6254509b2bc0 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:57.705589 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6490b03f-7ec5-4304-922b-6254509b2bc0 None admin] PATH_INFO: `/v3/users/1f4e105fbd3c43d4b25843da163b3dd3` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:57.706848 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6490b03f-7ec5-4304-922b-6254509b2bc0 None admin] RBAC: Authorizing `identity:get_user(user_id=1f4e105fbd3c43d4b25843da163b3dd3)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:57.708269 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6490b03f-7ec5-4304-922b-6254509b2bc0 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:57.709872 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 574/1127] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:32:57 2022] GET /identity/v3/users/1f4e105fbd3c43d4b25843da163b3dd3 => generated 286 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:32:57.715784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6165410-482f-4123-87ab-578c738980d3 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:57.811794 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6165410-482f-4123-87ab-578c738980d3 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:57.844212 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e8573f0-399d-45b9-9185-18138454c1fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:57.844802 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8573f0-399d-45b9-9185-18138454c1fe None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:57.845077 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8573f0-399d-45b9-9185-18138454c1fe None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:57.845347 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8573f0-399d-45b9-9185-18138454c1fe None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:57.863372 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:57.913071 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e8573f0-399d-45b9-9185-18138454c1fe None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:57.914578 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e8573f0-399d-45b9-9185-18138454c1fe None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:58.026013 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 554/1128] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:32:57 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 09:32:58.031911 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6490b03f-7ec5-4304-922b-6254509b2bc0 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:58.126965 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6490b03f-7ec5-4304-922b-6254509b2bc0 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:58.159578 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b6f96db-e179-4a67-a374-1b9329f894dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:58.160190 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6f96db-e179-4a67-a374-1b9329f894dc None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:58.160468 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6f96db-e179-4a67-a374-1b9329f894dc None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:58.160744 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6f96db-e179-4a67-a374-1b9329f894dc None admin] PATH_INFO: `/v3/projects/506899eec53d4333a54f3567f37027a4` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:58.161863 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b6f96db-e179-4a67-a374-1b9329f894dc None admin] RBAC: Authorizing `identity:get_project(project_id=506899eec53d4333a54f3567f37027a4)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:58.163287 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b6f96db-e179-4a67-a374-1b9329f894dc None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:58.164895 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 575/1129] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:32:58 2022] GET /identity/v3/projects/506899eec53d4333a54f3567f37027a4 => generated 311 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:32:58.171207 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e8573f0-399d-45b9-9185-18138454c1fe None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:32:58.219317 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:32:58.268737 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e8573f0-399d-45b9-9185-18138454c1fe None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:32:58.306363 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af99012b-6381-4b26-9e85-eb00dd11387b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:32:58.306950 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af99012b-6381-4b26-9e85-eb00dd11387b None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:32:58.307226 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af99012b-6381-4b26-9e85-eb00dd11387b None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:32:58.307497 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af99012b-6381-4b26-9e85-eb00dd11387b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:32:58.308225 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af99012b-6381-4b26-9e85-eb00dd11387b None admin] RBAC: Adding query filter params (role.id=d93b4c94b590457199c3d301f947b0fb, scope.project.id=506899eec53d4333a54f3567f37027a4, user.id=1f4e105fbd3c43d4b25843da163b3dd3) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:32:58.308531 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af99012b-6381-4b26-9e85-eb00dd11387b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:32:58.309971 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af99012b-6381-4b26-9e85-eb00dd11387b None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:32:58.316678 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 555/1130] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:32:58 2022] GET /identity/v3/role_assignments?user.id=1f4e105fbd3c43d4b25843da163b3dd3&scope.project.id=506899eec53d4333a54f3567f37027a4&role.id=d93b4c94b590457199c3d301f947b0fb => generated 608 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:33:00.479580 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195447fc-267e-4ff9-a2dd-20140d888812 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:00.479580 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195447fc-267e-4ff9-a2dd-20140d888812 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:00.480212 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195447fc-267e-4ff9-a2dd-20140d888812 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:00.480995 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 576/1131] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:33:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:33:00.486421 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f8aafd-5a47-4246-8d29-012a31e1a2a3 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:00.486826 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f8aafd-5a47-4246-8d29-012a31e1a2a3 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:00.486977 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f8aafd-5a47-4246-8d29-012a31e1a2a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:00.505493 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-38f8aafd-5a47-4246-8d29-012a31e1a2a3 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:33:00.507265 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-38f8aafd-5a47-4246-8d29-012a31e1a2a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:33:00.576344 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 556/1132] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:33:00 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:33:00.843248 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195447fc-267e-4ff9-a2dd-20140d888812 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:00.873573 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:33:00.944704 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195447fc-267e-4ff9-a2dd-20140d888812 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:00.978111 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d17becb-b302-44c9-a461-36041ad6582a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:00.978765 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d17becb-b302-44c9-a461-36041ad6582a None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:00.979046 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d17becb-b302-44c9-a461-36041ad6582a None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:00.979321 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d17becb-b302-44c9-a461-36041ad6582a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:00.980422 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d17becb-b302-44c9-a461-36041ad6582a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:33:00.981835 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d17becb-b302-44c9-a461-36041ad6582a None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:33:00.983443 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 577/1133] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:33:00 2022] GET /identity/v3/domains/default => generated 199 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:33:00.989304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f8aafd-5a47-4246-8d29-012a31e1a2a3 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:01.033578 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:33:01.086743 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f8aafd-5a47-4246-8d29-012a31e1a2a3 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:01.121098 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3918a659-4b4b-45e7-b658-cd83a59cb1f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:01.121624 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3918a659-4b4b-45e7-b658-cd83a59cb1f6 None admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:01.121908 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3918a659-4b4b-45e7-b658-cd83a59cb1f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:01.122185 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3918a659-4b4b-45e7-b658-cd83a59cb1f6 None admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:01.122868 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3918a659-4b4b-45e7-b658-cd83a59cb1f6 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:33:01.124301 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3918a659-4b4b-45e7-b658-cd83a59cb1f6 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:33:01.150991 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 557/1134] 10.222.0.25 () {60 vars in 1141 bytes} [Fri Oct 28 09:33:00 2022] POST /identity/v3/projects => generated 311 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:33:03.321306 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5648eb46-2948-4707-991b-783a42c13a4e None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:03.321868 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5648eb46-2948-4707-991b-783a42c13a4e None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:03.321868 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5648eb46-2948-4707-991b-783a42c13a4e None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:03.323212 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 578/1135] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:33:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:33:03.328487 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a1fd75e-a14e-470f-a30d-de398db49197 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:03.328885 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a1fd75e-a14e-470f-a30d-de398db49197 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:03.329074 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a1fd75e-a14e-470f-a30d-de398db49197 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:03.347472 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-9a1fd75e-a14e-470f-a30d-de398db49197 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:33:03.348655 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-9a1fd75e-a14e-470f-a30d-de398db49197 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:33:03.412838 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 558/1136] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:33:03 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:33:03.683056 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5648eb46-2948-4707-991b-783a42c13a4e None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:03.777339 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:33:03.782672 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5648eb46-2948-4707-991b-783a42c13a4e None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:03.815616 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495c7672-e7d9-474f-884c-bb21ffa3c73b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:03.816224 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495c7672-e7d9-474f-884c-bb21ffa3c73b None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:03.816501 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495c7672-e7d9-474f-884c-bb21ffa3c73b None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:03.816770 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495c7672-e7d9-474f-884c-bb21ffa3c73b None admin] PATH_INFO: `/v3/domains/default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:03.817868 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-495c7672-e7d9-474f-884c-bb21ffa3c73b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:33:03.819440 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-495c7672-e7d9-474f-884c-bb21ffa3c73b None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:33:03.820974 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 579/1137] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:33:03 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:33:03.826835 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a1fd75e-a14e-470f-a30d-de398db49197 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:03.853606 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:33:03.924528 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a1fd75e-a14e-470f-a30d-de398db49197 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:03.957279 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9db490b-82a9-4bd8-88ab-2f3ceb487565 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:03.957882 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9db490b-82a9-4bd8-88ab-2f3ceb487565 None admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:03.958166 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9db490b-82a9-4bd8-88ab-2f3ceb487565 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:03.958469 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9db490b-82a9-4bd8-88ab-2f3ceb487565 None admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:03.959173 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9db490b-82a9-4bd8-88ab-2f3ceb487565 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:33:03.960599 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9db490b-82a9-4bd8-88ab-2f3ceb487565 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:33:03.981501 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 559/1138] 10.222.0.25 () {60 vars in 1132 bytes} [Fri Oct 28 09:33:03 2022] POST /identity/v3/users => generated 286 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:33:06.358643 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6edb91-64c2-4b90-8887-29585a24671d None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:06.359239 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6edb91-64c2-4b90-8887-29585a24671d None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:06.359239 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6edb91-64c2-4b90-8887-29585a24671d None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:06.360506 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 580/1139] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:33:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:33:06.365903 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900c1dc3-3431-4a95-ae21-6543114e7f64 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:06.366337 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900c1dc3-3431-4a95-ae21-6543114e7f64 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:06.366495 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900c1dc3-3431-4a95-ae21-6543114e7f64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:06.384790 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-900c1dc3-3431-4a95-ae21-6543114e7f64 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:33:06.385983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-900c1dc3-3431-4a95-ae21-6543114e7f64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:33:06.450874 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 560/1140] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:33:06 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:33:06.718914 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6edb91-64c2-4b90-8887-29585a24671d None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:06.818542 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6edb91-64c2-4b90-8887-29585a24671d None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:06.852171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e505e96-8fae-4e0f-af49-a70eaa347d91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:06.852792 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e505e96-8fae-4e0f-af49-a70eaa347d91 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:06.853073 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e505e96-8fae-4e0f-af49-a70eaa347d91 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:06.853352 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e505e96-8fae-4e0f-af49-a70eaa347d91 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:06.863099 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e505e96-8fae-4e0f-af49-a70eaa347d91 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:33:06.868474 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-7e505e96-8fae-4e0f-af49-a70eaa347d91 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:33:06.868474 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:33:06.868474 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:33:06.868474 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:33:06.868474 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:33:06.868474 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:33:06.868474 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:33:06.868474 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:33:06.868474 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:33:06.868474 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:33:06.868474 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:33:06.868474 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:33:06.868474 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:33:06.868474 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:33:06.868474 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:33:06.868474 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:33:06.869663 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:33:06.869663 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:33:06.869663 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:33:06.869663 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:33:06.869663 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:33:06.869663 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:33:06.869663 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:33:06.869663 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:33:06.869663 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:33:06.869663 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:33:06.869663 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:33:06.869663 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:33:06.869663 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:33:06.869663 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:33:06.869663 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:33:06.869663 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:33:06.870803 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:33:06.870803 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:33:06.870803 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:33:06.870803 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:33:06.870803 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:33:06.870803 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:33:06.870803 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:33:06.870803 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:33:06.870803 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:33:06.870803 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:33:06.870803 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:33:06.870803 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:33:06.870803 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 581/1141] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Oct 28 09:33:06 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:33:06.875671 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-900c1dc3-3431-4a95-ae21-6543114e7f64 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:06.889717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:33:06.976018 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-900c1dc3-3431-4a95-ae21-6543114e7f64 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:07.009290 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405aa069-e5e0-4f00-9681-b2267b09dc1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:07.009893 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405aa069-e5e0-4f00-9681-b2267b09dc1a None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:07.010170 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405aa069-e5e0-4f00-9681-b2267b09dc1a None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:07.010466 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405aa069-e5e0-4f00-9681-b2267b09dc1a None admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:07.011120 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405aa069-e5e0-4f00-9681-b2267b09dc1a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:33:07.011422 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405aa069-e5e0-4f00-9681-b2267b09dc1a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:33:07.012777 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405aa069-e5e0-4f00-9681-b2267b09dc1a None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:33:07.020501 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 561/1142] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:33:06 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:33:07.026565 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e505e96-8fae-4e0f-af49-a70eaa347d91 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:07.054777 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:33:07.125260 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e505e96-8fae-4e0f-af49-a70eaa347d91 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:07.158667 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649cdd73-36b5-4b74-a790-c7dfa4e351cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:07.159532 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649cdd73-36b5-4b74-a790-c7dfa4e351cf None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:07.159813 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649cdd73-36b5-4b74-a790-c7dfa4e351cf None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:07.160086 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649cdd73-36b5-4b74-a790-c7dfa4e351cf None admin] PATH_INFO: `/v3/users/9d7851b54f354219a3f970bd8229103c` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:07.172623 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-649cdd73-36b5-4b74-a790-c7dfa4e351cf None admin] RBAC: Authorizing `identity:get_user(user_id=9d7851b54f354219a3f970bd8229103c)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:33:07.174036 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-649cdd73-36b5-4b74-a790-c7dfa4e351cf None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:33:07.175643 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 582/1143] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:33:07 2022] GET /identity/v3/users/9d7851b54f354219a3f970bd8229103c => generated 286 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:33:07.181516 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405aa069-e5e0-4f00-9681-b2267b09dc1a None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:07.254888 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:33:07.280812 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405aa069-e5e0-4f00-9681-b2267b09dc1a None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:07.313856 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c7e648-1ae4-4cee-8350-b7cdb9465e1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:07.314495 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c7e648-1ae4-4cee-8350-b7cdb9465e1a None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:07.314778 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c7e648-1ae4-4cee-8350-b7cdb9465e1a None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:07.315050 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c7e648-1ae4-4cee-8350-b7cdb9465e1a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:07.379640 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c7e648-1ae4-4cee-8350-b7cdb9465e1a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:33:07.381627 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c7e648-1ae4-4cee-8350-b7cdb9465e1a None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:33:07.452604 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:33:07.478082 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 562/1144] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:33:07 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 297 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 09:33:07.484054 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649cdd73-36b5-4b74-a790-c7dfa4e351cf None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:07.569822 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:33:07.586859 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649cdd73-36b5-4b74-a790-c7dfa4e351cf None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:07.620163 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee6bdb19-bc8c-4118-9b9a-151ad2b334f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:07.620790 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6bdb19-bc8c-4118-9b9a-151ad2b334f6 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:07.621068 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6bdb19-bc8c-4118-9b9a-151ad2b334f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:07.621341 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee6bdb19-bc8c-4118-9b9a-151ad2b334f6 None admin] PATH_INFO: `/v3/projects/07ca4d35046043d082eb060606890108` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:07.622464 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6bdb19-bc8c-4118-9b9a-151ad2b334f6 None admin] RBAC: Authorizing `identity:get_project(project_id=07ca4d35046043d082eb060606890108)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:33:07.623873 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee6bdb19-bc8c-4118-9b9a-151ad2b334f6 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:33:07.625488 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 583/1145] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:33:07 2022] GET /identity/v3/projects/07ca4d35046043d082eb060606890108 => generated 311 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:33:07.632155 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c7e648-1ae4-4cee-8350-b7cdb9465e1a None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:07.735614 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c7e648-1ae4-4cee-8350-b7cdb9465e1a None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:07.770350 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd22ad02-67f4-4236-9ce7-5546b2dd5b8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:07.771040 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd22ad02-67f4-4236-9ce7-5546b2dd5b8f None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:07.771323 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd22ad02-67f4-4236-9ce7-5546b2dd5b8f None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:07.771595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd22ad02-67f4-4236-9ce7-5546b2dd5b8f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:07.772314 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd22ad02-67f4-4236-9ce7-5546b2dd5b8f None admin] RBAC: Adding query filter params (role.id=e13d798a98ea4c3dbb87a236ec683d32, scope.project.id=07ca4d35046043d082eb060606890108, user.id=9d7851b54f354219a3f970bd8229103c) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:33:07.772621 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd22ad02-67f4-4236-9ce7-5546b2dd5b8f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:33:07.774039 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd22ad02-67f4-4236-9ce7-5546b2dd5b8f None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:33:07.779760 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 563/1146] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:33:07 2022] GET /identity/v3/role_assignments?user.id=9d7851b54f354219a3f970bd8229103c&scope.project.id=07ca4d35046043d082eb060606890108&role.id=e13d798a98ea4c3dbb87a236ec683d32 => generated 259 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:33:09.951187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab52cf0-d505-454b-a0e4-daf21062fc6a None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:09.951756 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab52cf0-d505-454b-a0e4-daf21062fc6a None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:09.951756 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab52cf0-d505-454b-a0e4-daf21062fc6a None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:09.953065 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 584/1147] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:33:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:33:09.958195 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bc8b4b-8aa8-4339-821c-7d9224d5d3cb None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:09.958678 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bc8b4b-8aa8-4339-821c-7d9224d5d3cb None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:09.958838 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34bc8b4b-8aa8-4339-821c-7d9224d5d3cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:09.976991 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-34bc8b4b-8aa8-4339-821c-7d9224d5d3cb None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:33:09.978178 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-34bc8b4b-8aa8-4339-821c-7d9224d5d3cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:33:10.007454 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:33:10.044281 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 564/1148] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:33:09 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 09:33:10.317441 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab52cf0-d505-454b-a0e4-daf21062fc6a None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:10.417080 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab52cf0-d505-454b-a0e4-daf21062fc6a None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:10.450543 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3165cdcf-d2f5-4971-8515-451f3de8e6f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:10.451168 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3165cdcf-d2f5-4971-8515-451f3de8e6f9 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:10.451451 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3165cdcf-d2f5-4971-8515-451f3de8e6f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:10.451728 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3165cdcf-d2f5-4971-8515-451f3de8e6f9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:10.462038 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3165cdcf-d2f5-4971-8515-451f3de8e6f9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:33:10.466842 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-3165cdcf-d2f5-4971-8515-451f3de8e6f9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:33:10.466842 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:33:10.466842 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:33:10.466842 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:33:10.466842 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:33:10.466842 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:33:10.466842 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:33:10.466842 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:33:10.466842 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:33:10.466842 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:33:10.466842 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:33:10.466842 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:33:10.466842 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:33:10.466842 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:33:10.466842 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:33:10.466842 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:33:10.467788 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:33:10.467788 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:33:10.467788 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:33:10.467788 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:33:10.467788 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:33:10.467788 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:33:10.467788 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:33:10.467788 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:33:10.467788 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:33:10.467788 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:33:10.467788 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:33:10.467788 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:33:10.467788 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:33:10.467788 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:33:10.467788 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:33:10.467788 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:33:10.468643 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:33:10.468643 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:33:10.468643 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:33:10.468643 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:33:10.468643 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:33:10.468643 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:33:10.468643 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:33:10.468643 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:33:10.468643 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:33:10.468643 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:33:10.468643 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:33:10.468643 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:33:10.468643 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 585/1149] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Oct 28 09:33:10 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:33:10.474043 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34bc8b4b-8aa8-4339-821c-7d9224d5d3cb None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:10.574752 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34bc8b4b-8aa8-4339-821c-7d9224d5d3cb None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:10.608298 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-846ab33e-4d84-444f-ad59-599c755b7741 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:10.608906 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846ab33e-4d84-444f-ad59-599c755b7741 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:10.609192 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846ab33e-4d84-444f-ad59-599c755b7741 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:10.609471 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846ab33e-4d84-444f-ad59-599c755b7741 None admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:10.610133 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-846ab33e-4d84-444f-ad59-599c755b7741 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:33:10.610467 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-846ab33e-4d84-444f-ad59-599c755b7741 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:33:10.611841 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-846ab33e-4d84-444f-ad59-599c755b7741 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:33:10.620280 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 565/1150] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:33:10 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:33:10.626356 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3165cdcf-d2f5-4971-8515-451f3de8e6f9 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:10.641360 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:33:10.723059 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3165cdcf-d2f5-4971-8515-451f3de8e6f9 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:10.756472 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75aea9d8-b609-489e-80c3-9d4624fee601 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:10.757089 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75aea9d8-b609-489e-80c3-9d4624fee601 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:10.757369 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75aea9d8-b609-489e-80c3-9d4624fee601 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:10.757643 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75aea9d8-b609-489e-80c3-9d4624fee601 None admin] PATH_INFO: `/v3/users/9d7851b54f354219a3f970bd8229103c` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:10.758773 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75aea9d8-b609-489e-80c3-9d4624fee601 None admin] RBAC: Authorizing `identity:get_user(user_id=9d7851b54f354219a3f970bd8229103c)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:33:10.760174 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75aea9d8-b609-489e-80c3-9d4624fee601 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:33:10.761722 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 586/1151] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:33:10 2022] GET /identity/v3/users/9d7851b54f354219a3f970bd8229103c => generated 286 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:33:10.767572 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-846ab33e-4d84-444f-ad59-599c755b7741 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:10.797856 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:33:10.865686 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-846ab33e-4d84-444f-ad59-599c755b7741 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:10.899657 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c732e2d-c587-4e9f-94ad-30201dd0d14e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:10.900295 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c732e2d-c587-4e9f-94ad-30201dd0d14e None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:10.900578 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c732e2d-c587-4e9f-94ad-30201dd0d14e None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:10.900856 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c732e2d-c587-4e9f-94ad-30201dd0d14e None admin] PATH_INFO: `/v3/projects/07ca4d35046043d082eb060606890108` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:10.901979 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c732e2d-c587-4e9f-94ad-30201dd0d14e None admin] RBAC: Authorizing `identity:get_project(project_id=07ca4d35046043d082eb060606890108)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:33:10.903451 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c732e2d-c587-4e9f-94ad-30201dd0d14e None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:33:10.905082 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 566/1152] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:33:10 2022] GET /identity/v3/projects/07ca4d35046043d082eb060606890108 => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:33:10.911555 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75aea9d8-b609-489e-80c3-9d4624fee601 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:10.991597 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:33:11.011071 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75aea9d8-b609-489e-80c3-9d4624fee601 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:11.043571 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1af9f1-9fc2-4bd8-96c3-62eaf90ded45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:11.044225 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1af9f1-9fc2-4bd8-96c3-62eaf90ded45 None admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:11.044506 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1af9f1-9fc2-4bd8-96c3-62eaf90ded45 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:11.044783 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1af9f1-9fc2-4bd8-96c3-62eaf90ded45 None admin] PATH_INFO: `/v3/projects/07ca4d35046043d082eb060606890108/users/9d7851b54f354219a3f970bd8229103c/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:11.046685 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd1af9f1-9fc2-4bd8-96c3-62eaf90ded45 None admin] RBAC: Authorizing `identity:create_grant(project_id=07ca4d35046043d082eb060606890108, user_id=9d7851b54f354219a3f970bd8229103c, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:33:11.048119 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd1af9f1-9fc2-4bd8-96c3-62eaf90ded45 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:33:11.061530 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 587/1153] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Oct 28 09:33:10 2022] PUT /identity/v3/projects/07ca4d35046043d082eb060606890108/users/9d7851b54f354219a3f970bd8229103c/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:33:13.277319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521fa4b5-f589-440e-92c4-0d2a35582d2b None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:13.277897 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521fa4b5-f589-440e-92c4-0d2a35582d2b None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:13.278299 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521fa4b5-f589-440e-92c4-0d2a35582d2b None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:13.280768 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 567/1154] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:33:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:33:13.287100 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037e78e9-e071-484f-83cd-6e31f123380f None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:13.287649 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037e78e9-e071-484f-83cd-6e31f123380f None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:13.287649 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-037e78e9-e071-484f-83cd-6e31f123380f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:13.307563 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-037e78e9-e071-484f-83cd-6e31f123380f None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:33:13.308783 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-037e78e9-e071-484f-83cd-6e31f123380f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:33:13.380372 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 588/1155] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:33:13 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:33:13.650578 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521fa4b5-f589-440e-92c4-0d2a35582d2b None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:13.755224 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:33:13.760871 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521fa4b5-f589-440e-92c4-0d2a35582d2b None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:13.799160 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bcb1552-e827-43d9-826d-16053e120719 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:13.799795 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcb1552-e827-43d9-826d-16053e120719 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:13.800079 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcb1552-e827-43d9-826d-16053e120719 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:13.800355 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bcb1552-e827-43d9-826d-16053e120719 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:13.810748 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bcb1552-e827-43d9-826d-16053e120719 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:33:13.815672 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-0bcb1552-e827-43d9-826d-16053e120719 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:33:13.815672 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:33:13.815672 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:33:13.815672 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:33:13.815672 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:33:13.815672 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:33:13.815672 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:33:13.815672 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:33:13.815672 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:33:13.815672 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:33:13.815672 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:33:13.815672 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:33:13.815672 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:33:13.815672 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:33:13.815672 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:33:13.815672 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:33:13.817050 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:33:13.817050 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:33:13.817050 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:33:13.817050 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:33:13.817050 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:33:13.817050 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:33:13.817050 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:33:13.817050 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:33:13.817050 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:33:13.817050 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:33:13.817050 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:33:13.817050 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:33:13.817050 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:33:13.817050 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:33:13.817050 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:33:13.817050 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:33:13.818545 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:33:13.818545 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:33:13.818545 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:33:13.818545 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:33:13.818545 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:33:13.818545 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:33:13.818545 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:33:13.818545 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:33:13.818545 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:33:13.818545 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:33:13.818545 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:33:13.818545 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:33:13.818545 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 568/1156] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Oct 28 09:33:13 2022] GET /identity/v3/roles/admin => generated 83 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:33:13.823423 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-037e78e9-e071-484f-83cd-6e31f123380f None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:13.894353 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:33:13.927174 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-037e78e9-e071-484f-83cd-6e31f123380f None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:13.961421 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbdda0f3-822b-4969-90f1-ed86cea6282a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:13.962034 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbdda0f3-822b-4969-90f1-ed86cea6282a None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:13.962318 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbdda0f3-822b-4969-90f1-ed86cea6282a None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:13.962620 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbdda0f3-822b-4969-90f1-ed86cea6282a None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:13.963283 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbdda0f3-822b-4969-90f1-ed86cea6282a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:33:13.963590 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbdda0f3-822b-4969-90f1-ed86cea6282a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:33:13.964957 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbdda0f3-822b-4969-90f1-ed86cea6282a None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:33:13.973168 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 589/1157] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:33:13 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:33:13.979247 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bcb1552-e827-43d9-826d-16053e120719 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:14.079683 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bcb1552-e827-43d9-826d-16053e120719 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:14.113189 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-997d6a3e-52ea-4956-b2d5-9e91eb2167c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:14.113816 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997d6a3e-52ea-4956-b2d5-9e91eb2167c2 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:14.114093 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997d6a3e-52ea-4956-b2d5-9e91eb2167c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:14.114361 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-997d6a3e-52ea-4956-b2d5-9e91eb2167c2 None admin] PATH_INFO: `/v3/users/9d7851b54f354219a3f970bd8229103c` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:14.116023 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-997d6a3e-52ea-4956-b2d5-9e91eb2167c2 None admin] RBAC: Authorizing `identity:get_user(user_id=9d7851b54f354219a3f970bd8229103c)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:33:14.117445 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-997d6a3e-52ea-4956-b2d5-9e91eb2167c2 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:33:14.119046 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 569/1158] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:33:13 2022] GET /identity/v3/users/9d7851b54f354219a3f970bd8229103c => generated 286 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:33:14.124952 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbdda0f3-822b-4969-90f1-ed86cea6282a None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:14.228058 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbdda0f3-822b-4969-90f1-ed86cea6282a None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:14.261364 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:33:14.267117 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4fab708-c72f-465f-adf2-16f936d80125 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:14.267729 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4fab708-c72f-465f-adf2-16f936d80125 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:14.268005 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4fab708-c72f-465f-adf2-16f936d80125 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:14.268274 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4fab708-c72f-465f-adf2-16f936d80125 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:14.336674 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4fab708-c72f-465f-adf2-16f936d80125 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:33:14.338121 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4fab708-c72f-465f-adf2-16f936d80125 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:33:14.438360 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 590/1159] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:33:14 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 09:33:14.444410 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-997d6a3e-52ea-4956-b2d5-9e91eb2167c2 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:14.471685 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:33:14.543417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-997d6a3e-52ea-4956-b2d5-9e91eb2167c2 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:14.577909 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f76173e-b51d-44f0-9a0c-5f28b106ba48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:14.578575 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f76173e-b51d-44f0-9a0c-5f28b106ba48 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:14.578859 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f76173e-b51d-44f0-9a0c-5f28b106ba48 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:14.579138 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f76173e-b51d-44f0-9a0c-5f28b106ba48 None admin] PATH_INFO: `/v3/projects/07ca4d35046043d082eb060606890108` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:14.580258 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f76173e-b51d-44f0-9a0c-5f28b106ba48 None admin] RBAC: Authorizing `identity:get_project(project_id=07ca4d35046043d082eb060606890108)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:33:14.581690 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f76173e-b51d-44f0-9a0c-5f28b106ba48 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:33:14.583584 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 570/1160] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:33:14 2022] GET /identity/v3/projects/07ca4d35046043d082eb060606890108 => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:33:14.589420 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4fab708-c72f-465f-adf2-16f936d80125 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:14.619808 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:33:14.687567 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4fab708-c72f-465f-adf2-16f936d80125 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:14.729744 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88353b1-22eb-499a-a4f2-fba12864e652 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:14.730532 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88353b1-22eb-499a-a4f2-fba12864e652 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:14.730821 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88353b1-22eb-499a-a4f2-fba12864e652 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:14.731098 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88353b1-22eb-499a-a4f2-fba12864e652 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:14.731814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d88353b1-22eb-499a-a4f2-fba12864e652 None admin] RBAC: Adding query filter params (role.id=e13d798a98ea4c3dbb87a236ec683d32, scope.project.id=07ca4d35046043d082eb060606890108, user.id=9d7851b54f354219a3f970bd8229103c) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:33:14.732125 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d88353b1-22eb-499a-a4f2-fba12864e652 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:33:14.733543 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d88353b1-22eb-499a-a4f2-fba12864e652 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:33:14.739381 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 591/1161] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:33:14 2022] GET /identity/v3/role_assignments?user.id=9d7851b54f354219a3f970bd8229103c&scope.project.id=07ca4d35046043d082eb060606890108&role.id=e13d798a98ea4c3dbb87a236ec683d32 => generated 608 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:33:16.904644 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0a2194-4146-404e-9a3b-367c8de95b71 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:16.905209 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0a2194-4146-404e-9a3b-367c8de95b71 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:16.905209 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0a2194-4146-404e-9a3b-367c8de95b71 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:16.906725 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 571/1162] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:33:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:33:16.912892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24839660-30d0-47af-baff-e04f0da4b0a5 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:16.913314 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24839660-30d0-47af-baff-e04f0da4b0a5 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:16.913428 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24839660-30d0-47af-baff-e04f0da4b0a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:16.932022 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-24839660-30d0-47af-baff-e04f0da4b0a5 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:33:16.933209 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-24839660-30d0-47af-baff-e04f0da4b0a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:33:16.998377 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 592/1163] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:33:16 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 09:33:17.266176 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0a2194-4146-404e-9a3b-367c8de95b71 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:17.358794 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:33:17.366832 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0a2194-4146-404e-9a3b-367c8de95b71 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:17.400233 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275ae079-2cb6-4a20-9c13-44c2cee4e952 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:17.400836 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275ae079-2cb6-4a20-9c13-44c2cee4e952 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:17.401114 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275ae079-2cb6-4a20-9c13-44c2cee4e952 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:17.401386 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275ae079-2cb6-4a20-9c13-44c2cee4e952 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:17.466810 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275ae079-2cb6-4a20-9c13-44c2cee4e952 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:33:17.468244 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275ae079-2cb6-4a20-9c13-44c2cee4e952 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:33:17.557721 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:33:17.566832 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 572/1164] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:33:17 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 09:33:17.573080 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24839660-30d0-47af-baff-e04f0da4b0a5 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:17.599545 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:33:17.668931 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24839660-30d0-47af-baff-e04f0da4b0a5 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:17.701838 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b56b7f6-6c3c-4be3-8376-ca767c83920c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:17.702490 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b56b7f6-6c3c-4be3-8376-ca767c83920c None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:17.702771 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b56b7f6-6c3c-4be3-8376-ca767c83920c None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:17.703045 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b56b7f6-6c3c-4be3-8376-ca767c83920c None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:17.709782 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b56b7f6-6c3c-4be3-8376-ca767c83920c None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:33:17.711224 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b56b7f6-6c3c-4be3-8376-ca767c83920c None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:33:17.722305 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-5b56b7f6-6c3c-4be3-8376-ca767c83920c None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Oct 28 09:33:17.722305 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:33:17.722305 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Oct 28 09:33:17.722305 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Oct 28 09:33:17.722305 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 09:33:17.722305 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 09:33:17.722305 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 09:33:17.722305 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 09:33:17.722305 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Oct 28 09:33:17.722305 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:33:17.722305 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 09:33:17.722305 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:33:17.722305 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:33:17.722305 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:33:17.722305 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:33:17.722305 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:33:17.722305 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:33:17.723540 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:33:17.723540 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:33:17.723540 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:33:17.723540 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:33:17.723540 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:33:17.723540 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:33:17.723540 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:33:17.723540 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:33:17.723540 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:33:17.723540 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:33:17.723540 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:33:17.723540 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:33:17.723540 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:33:17.723540 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:33:17.723540 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:33:17.723540 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:33:17.724467 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:33:17.724467 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:33:17.724467 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:33:17.724467 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:33:17.724467 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:33:17.724467 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:33:17.724467 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:33:17.724467 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:33:17.724467 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:33:17.724467 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:33:17.724467 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Oct 28 09:33:17.724467 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:33:17.724467 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Oct 28 09:33:17.724467 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:33:17.724467 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 593/1165] 10.222.0.25 () {56 vars in 1117 bytes} [Fri Oct 28 09:33:17 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:33:17.729698 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275ae079-2cb6-4a20-9c13-44c2cee4e952 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:17.826358 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275ae079-2cb6-4a20-9c13-44c2cee4e952 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:17.858932 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-176c277e-9d6c-4856-a9af-cf79206e9433 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:17.859575 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176c277e-9d6c-4856-a9af-cf79206e9433 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:17.859887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176c277e-9d6c-4856-a9af-cf79206e9433 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:17.860161 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-176c277e-9d6c-4856-a9af-cf79206e9433 None admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:17.860799 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-176c277e-9d6c-4856-a9af-cf79206e9433 None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:33:17.861099 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-176c277e-9d6c-4856-a9af-cf79206e9433 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:33:17.862720 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-176c277e-9d6c-4856-a9af-cf79206e9433 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:33:17.870567 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 573/1166] 10.222.0.25 () {56 vars in 1115 bytes} [Fri Oct 28 09:33:17 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:33:20.018786 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cbb06d-04c1-4b04-8653-065e853bf8d9 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:20.019317 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cbb06d-04c1-4b04-8653-065e853bf8d9 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:20.019317 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77cbb06d-04c1-4b04-8653-065e853bf8d9 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:20.021282 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 594/1167] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:33:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:33:20.026613 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ac7079-5a3f-496a-bedc-567f75f3a9d3 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:20.027123 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ac7079-5a3f-496a-bedc-567f75f3a9d3 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:20.027249 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82ac7079-5a3f-496a-bedc-567f75f3a9d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:20.045209 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-82ac7079-5a3f-496a-bedc-567f75f3a9d3 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:33:20.046421 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-82ac7079-5a3f-496a-bedc-567f75f3a9d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:33:20.097898 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:33:20.112253 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 574/1168] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:33:20 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 09:33:20.380940 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77cbb06d-04c1-4b04-8653-065e853bf8d9 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:20.468327 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:33:20.484892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77cbb06d-04c1-4b04-8653-065e853bf8d9 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:20.517509 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b0a889-6e71-4c00-a17e-3ffea943c3cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:20.518105 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b0a889-6e71-4c00-a17e-3ffea943c3cb None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:20.518384 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b0a889-6e71-4c00-a17e-3ffea943c3cb None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:20.518684 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b0a889-6e71-4c00-a17e-3ffea943c3cb None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:20.519349 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b0a889-6e71-4c00-a17e-3ffea943c3cb None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:33:20.520717 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b0a889-6e71-4c00-a17e-3ffea943c3cb None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:33:20.536292 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 595/1169] 10.222.0.25 () {60 vars in 1138 bytes} [Fri Oct 28 09:33:20 2022] POST /identity/v3/domains => generated 267 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:33:22.707309 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc7c48e-86c6-4df0-b482-c052088333cf None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:22.707831 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc7c48e-86c6-4df0-b482-c052088333cf None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:22.707831 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc7c48e-86c6-4df0-b482-c052088333cf None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:22.709237 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 575/1170] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:33:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:33:22.714350 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc2da74-17f1-47e2-832b-1980f34fa8a4 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:22.714753 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc2da74-17f1-47e2-832b-1980f34fa8a4 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:22.714939 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc2da74-17f1-47e2-832b-1980f34fa8a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:22.733092 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-3dc2da74-17f1-47e2-832b-1980f34fa8a4 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:33:22.734276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-3dc2da74-17f1-47e2-832b-1980f34fa8a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:33:22.798526 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 596/1171] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:33:22 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:33:23.068494 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc7c48e-86c6-4df0-b482-c052088333cf None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:23.168811 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc7c48e-86c6-4df0-b482-c052088333cf None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:23.201848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f83aee-cd23-437c-a393-bc582cd83e15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:23.202492 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f83aee-cd23-437c-a393-bc582cd83e15 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:23.202777 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f83aee-cd23-437c-a393-bc582cd83e15 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:23.203054 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f83aee-cd23-437c-a393-bc582cd83e15 None admin] PATH_INFO: `/v3/domains/9e56f843bddc4539acb9d6b91d81306f` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:23.213523 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f83aee-cd23-437c-a393-bc582cd83e15 None admin] RBAC: Authorizing `identity:get_domain(domain_id=9e56f843bddc4539acb9d6b91d81306f)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:33:23.214998 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f83aee-cd23-437c-a393-bc582cd83e15 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:33:23.216568 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 576/1172] 10.222.0.25 () {56 vars in 1183 bytes} [Fri Oct 28 09:33:23 2022] GET /identity/v3/domains/9e56f843bddc4539acb9d6b91d81306f => generated 267 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:33:23.222483 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dc2da74-17f1-47e2-832b-1980f34fa8a4 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:23.297605 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:33:23.321324 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dc2da74-17f1-47e2-832b-1980f34fa8a4 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:23.355146 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81671f87-c502-4cc0-a97f-5593d87f9de4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:23.355743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81671f87-c502-4cc0-a97f-5593d87f9de4 None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:23.356023 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81671f87-c502-4cc0-a97f-5593d87f9de4 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:23.356292 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81671f87-c502-4cc0-a97f-5593d87f9de4 None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:23.356930 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81671f87-c502-4cc0-a97f-5593d87f9de4 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:33:23.358340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81671f87-c502-4cc0-a97f-5593d87f9de4 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:33:23.387662 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 597/1173] 10.222.0.25 () {60 vars in 1141 bytes} [Fri Oct 28 09:33:23 2022] POST /identity/v3/projects => generated 361 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:33:25.587258 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442b01fd-ab86-4896-b601-473be9643fd3 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:25.587673 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442b01fd-ab86-4896-b601-473be9643fd3 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:25.587822 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442b01fd-ab86-4896-b601-473be9643fd3 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:25.589200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 577/1174] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:33:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:33:25.594502 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22911985-f1e2-42b4-ab29-a8f6b66fad1d None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:25.594914 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22911985-f1e2-42b4-ab29-a8f6b66fad1d None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:25.595057 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22911985-f1e2-42b4-ab29-a8f6b66fad1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:25.613386 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-22911985-f1e2-42b4-ab29-a8f6b66fad1d None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:33:25.614644 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-22911985-f1e2-42b4-ab29-a8f6b66fad1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:33:25.681489 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 598/1175] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:33:25 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 09:33:25.956510 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442b01fd-ab86-4896-b601-473be9643fd3 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:26.006921 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:33:26.058923 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442b01fd-ab86-4896-b601-473be9643fd3 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:26.091991 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da126bd3-de2d-4f3b-9595-c2619aafdd97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:26.092625 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da126bd3-de2d-4f3b-9595-c2619aafdd97 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:26.092903 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da126bd3-de2d-4f3b-9595-c2619aafdd97 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:26.093176 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da126bd3-de2d-4f3b-9595-c2619aafdd97 None admin] PATH_INFO: `/v3/domains/9e56f843bddc4539acb9d6b91d81306f` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:26.094277 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da126bd3-de2d-4f3b-9595-c2619aafdd97 None admin] RBAC: Authorizing `identity:get_domain(domain_id=9e56f843bddc4539acb9d6b91d81306f)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:33:26.095766 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da126bd3-de2d-4f3b-9595-c2619aafdd97 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:33:26.097489 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 578/1176] 10.222.0.25 () {56 vars in 1183 bytes} [Fri Oct 28 09:33:25 2022] GET /identity/v3/domains/9e56f843bddc4539acb9d6b91d81306f => generated 267 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:33:26.105084 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22911985-f1e2-42b4-ab29-a8f6b66fad1d None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:26.163145 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:33:26.203997 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22911985-f1e2-42b4-ab29-a8f6b66fad1d None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:26.236427 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22613c82-11f0-4e47-b004-1796df43c0b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:26.236962 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22613c82-11f0-4e47-b004-1796df43c0b1 None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:26.237243 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22613c82-11f0-4e47-b004-1796df43c0b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:26.237516 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22613c82-11f0-4e47-b004-1796df43c0b1 None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:26.238158 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22613c82-11f0-4e47-b004-1796df43c0b1 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:33:26.240101 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22613c82-11f0-4e47-b004-1796df43c0b1 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:33:26.260443 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 599/1177] 10.222.0.25 () {60 vars in 1132 bytes} [Fri Oct 28 09:33:26 2022] POST /identity/v3/users => generated 311 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:33:28.476472 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435c77be-0c35-4da4-b24f-844b80fc0a79 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:28.477066 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435c77be-0c35-4da4-b24f-844b80fc0a79 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:28.477066 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435c77be-0c35-4da4-b24f-844b80fc0a79 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:28.478376 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 579/1178] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:33:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:33:28.484548 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e96701-b2ed-4189-8df9-d21f539e5544 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:28.485075 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e96701-b2ed-4189-8df9-d21f539e5544 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:28.485157 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e96701-b2ed-4189-8df9-d21f539e5544 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:28.506340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-c5e96701-b2ed-4189-8df9-d21f539e5544 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:33:28.507598 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-c5e96701-b2ed-4189-8df9-d21f539e5544 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:33:28.582693 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 600/1179] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:33:28 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:33:28.861230 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435c77be-0c35-4da4-b24f-844b80fc0a79 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:28.960915 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435c77be-0c35-4da4-b24f-844b80fc0a79 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:28.978444 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:33:28.996518 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-754e2194-6c82-4704-bb63-808d7eaabddc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:28.997141 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754e2194-6c82-4704-bb63-808d7eaabddc None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:28.997421 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754e2194-6c82-4704-bb63-808d7eaabddc None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:28.997693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-754e2194-6c82-4704-bb63-808d7eaabddc None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:29.007665 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-754e2194-6c82-4704-bb63-808d7eaabddc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:33:29.012406 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-754e2194-6c82-4704-bb63-808d7eaabddc None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:33:29.012406 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:33:29.012406 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:33:29.012406 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:33:29.012406 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:33:29.012406 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:33:29.012406 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:33:29.012406 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:33:29.012406 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:33:29.012406 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:33:29.012406 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:33:29.012406 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:33:29.012406 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:33:29.012406 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:33:29.012406 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:33:29.012406 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:33:29.013475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:33:29.013475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:33:29.013475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:33:29.013475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:33:29.013475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:33:29.013475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:33:29.013475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:33:29.013475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:33:29.013475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:33:29.013475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:33:29.013475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:33:29.013475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:33:29.013475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:33:29.013475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:33:29.013475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:33:29.013475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:33:29.014355 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:33:29.014355 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:33:29.014355 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:33:29.014355 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:33:29.014355 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:33:29.014355 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:33:29.014355 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:33:29.014355 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:33:29.014355 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:33:29.014355 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:33:29.014355 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:33:29.014355 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:33:29.014355 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 580/1180] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Oct 28 09:33:28 2022] GET /identity/v3/roles/admin => generated 83 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:33:29.019621 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e96701-b2ed-4189-8df9-d21f539e5544 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:29.065296 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:33:29.120579 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e96701-b2ed-4189-8df9-d21f539e5544 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:29.153654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34eee32-1c17-48eb-989a-74ee820790b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:29.154261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34eee32-1c17-48eb-989a-74ee820790b5 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:29.154578 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34eee32-1c17-48eb-989a-74ee820790b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:29.154855 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34eee32-1c17-48eb-989a-74ee820790b5 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:29.155513 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f34eee32-1c17-48eb-989a-74ee820790b5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:33:29.155820 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f34eee32-1c17-48eb-989a-74ee820790b5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:33:29.157174 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f34eee32-1c17-48eb-989a-74ee820790b5 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:33:29.165516 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 601/1181] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:33:29 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:33:29.171719 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-754e2194-6c82-4704-bb63-808d7eaabddc None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:29.268697 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-754e2194-6c82-4704-bb63-808d7eaabddc None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:29.302378 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01144c17-a6d8-4b23-bd94-423e4ef29f13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:29.303061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01144c17-a6d8-4b23-bd94-423e4ef29f13 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:29.303343 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01144c17-a6d8-4b23-bd94-423e4ef29f13 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:29.303619 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01144c17-a6d8-4b23-bd94-423e4ef29f13 None admin] PATH_INFO: `/v3/users/9fc94d65946c4437aa9b1a9d770bd58c` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:29.316570 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01144c17-a6d8-4b23-bd94-423e4ef29f13 None admin] RBAC: Authorizing `identity:get_user(user_id=9fc94d65946c4437aa9b1a9d770bd58c)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:33:29.317994 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01144c17-a6d8-4b23-bd94-423e4ef29f13 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:33:29.319632 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 581/1182] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:33:29 2022] GET /identity/v3/users/9fc94d65946c4437aa9b1a9d770bd58c => generated 311 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:33:29.325502 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34eee32-1c17-48eb-989a-74ee820790b5 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:29.424940 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34eee32-1c17-48eb-989a-74ee820790b5 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:29.432068 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:33:29.460203 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21db0fee-6d30-4a53-bf53-6c37724984e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:29.460808 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21db0fee-6d30-4a53-bf53-6c37724984e8 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:29.461085 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21db0fee-6d30-4a53-bf53-6c37724984e8 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:29.461359 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21db0fee-6d30-4a53-bf53-6c37724984e8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:29.525468 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21db0fee-6d30-4a53-bf53-6c37724984e8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:33:29.526916 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21db0fee-6d30-4a53-bf53-6c37724984e8 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:33:29.621162 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 602/1183] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:33:29 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 296 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 09:33:29.627156 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01144c17-a6d8-4b23-bd94-423e4ef29f13 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:29.661012 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:33:29.728055 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01144c17-a6d8-4b23-bd94-423e4ef29f13 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:29.760706 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25545c92-fe38-442d-84a6-37867399bc8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:29.761328 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25545c92-fe38-442d-84a6-37867399bc8a None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:29.761613 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25545c92-fe38-442d-84a6-37867399bc8a None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:29.761888 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25545c92-fe38-442d-84a6-37867399bc8a None admin] PATH_INFO: `/v3/projects/4fc6cd78acc4462f9fe5808abc6782e1` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:29.763031 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25545c92-fe38-442d-84a6-37867399bc8a None admin] RBAC: Authorizing `identity:get_project(project_id=4fc6cd78acc4462f9fe5808abc6782e1)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:33:29.764459 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25545c92-fe38-442d-84a6-37867399bc8a None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:33:29.766081 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 582/1184] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:33:29 2022] GET /identity/v3/projects/4fc6cd78acc4462f9fe5808abc6782e1 => generated 361 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:33:29.772801 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21db0fee-6d30-4a53-bf53-6c37724984e8 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:29.781007 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:33:29.869495 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21db0fee-6d30-4a53-bf53-6c37724984e8 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:29.901958 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b588ab4-7b4f-4524-8ef9-990296e921e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:29.902581 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b588ab4-7b4f-4524-8ef9-990296e921e3 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:29.902863 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b588ab4-7b4f-4524-8ef9-990296e921e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:29.903133 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b588ab4-7b4f-4524-8ef9-990296e921e3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:29.903841 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b588ab4-7b4f-4524-8ef9-990296e921e3 None admin] RBAC: Adding query filter params (role.id=e13d798a98ea4c3dbb87a236ec683d32, scope.project.id=4fc6cd78acc4462f9fe5808abc6782e1, user.id=9fc94d65946c4437aa9b1a9d770bd58c) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:33:29.904147 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b588ab4-7b4f-4524-8ef9-990296e921e3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:33:29.905538 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b588ab4-7b4f-4524-8ef9-990296e921e3 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:33:29.911039 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 603/1185] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:33:29 2022] GET /identity/v3/role_assignments?user.id=9fc94d65946c4437aa9b1a9d770bd58c&scope.project.id=4fc6cd78acc4462f9fe5808abc6782e1&role.id=e13d798a98ea4c3dbb87a236ec683d32 => generated 259 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:33:32.052347 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88388090-1ce2-4cbd-b360-211ca62feb1b None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:32.052912 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88388090-1ce2-4cbd-b360-211ca62feb1b None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:32.052912 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88388090-1ce2-4cbd-b360-211ca62feb1b None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:32.054233 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 583/1186] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:33:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:33:32.059613 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515ed5b8-19b1-4e75-8bea-fe85f123de59 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:32.060029 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515ed5b8-19b1-4e75-8bea-fe85f123de59 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:32.060173 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-515ed5b8-19b1-4e75-8bea-fe85f123de59 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:32.078495 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-515ed5b8-19b1-4e75-8bea-fe85f123de59 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:33:32.079680 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-515ed5b8-19b1-4e75-8bea-fe85f123de59 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:33:32.140535 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:33:32.145493 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 604/1187] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:33:32 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:33:32.417454 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88388090-1ce2-4cbd-b360-211ca62feb1b None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:32.516215 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88388090-1ce2-4cbd-b360-211ca62feb1b None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:32.517710 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:33:32.550754 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cee8a94-cbd8-4597-8a3d-2c155e93b8ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:32.551384 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cee8a94-cbd8-4597-8a3d-2c155e93b8ff None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:32.551665 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cee8a94-cbd8-4597-8a3d-2c155e93b8ff None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:32.551995 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cee8a94-cbd8-4597-8a3d-2c155e93b8ff None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:32.563317 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cee8a94-cbd8-4597-8a3d-2c155e93b8ff None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:33:32.568682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-3cee8a94-cbd8-4597-8a3d-2c155e93b8ff None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:33:32.568682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:33:32.568682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:33:32.568682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:33:32.568682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:33:32.568682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:33:32.568682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:33:32.568682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:33:32.568682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:33:32.568682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:33:32.568682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:33:32.568682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:33:32.568682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:33:32.568682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:33:32.568682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:33:32.568682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:33:32.569767 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:33:32.569767 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:33:32.569767 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:33:32.569767 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:33:32.569767 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:33:32.569767 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:33:32.569767 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:33:32.569767 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:33:32.569767 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:33:32.569767 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:33:32.569767 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:33:32.569767 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:33:32.569767 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:33:32.569767 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:33:32.569767 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:33:32.569767 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:33:32.570788 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:33:32.570788 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:33:32.570788 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:33:32.570788 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:33:32.570788 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:33:32.570788 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:33:32.570788 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:33:32.570788 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:33:32.570788 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:33:32.570788 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:33:32.570788 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:33:32.570788 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:33:32.570788 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 584/1188] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Oct 28 09:33:32 2022] GET /identity/v3/roles/admin => generated 83 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:33:32.575891 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515ed5b8-19b1-4e75-8bea-fe85f123de59 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:32.672853 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-515ed5b8-19b1-4e75-8bea-fe85f123de59 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:32.705890 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0292bc69-53ea-4fc8-b9a2-b458c00daf91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:32.706524 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0292bc69-53ea-4fc8-b9a2-b458c00daf91 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:32.706808 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0292bc69-53ea-4fc8-b9a2-b458c00daf91 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:32.707084 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0292bc69-53ea-4fc8-b9a2-b458c00daf91 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:32.707745 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0292bc69-53ea-4fc8-b9a2-b458c00daf91 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:33:32.708069 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0292bc69-53ea-4fc8-b9a2-b458c00daf91 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:33:32.709957 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0292bc69-53ea-4fc8-b9a2-b458c00daf91 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:33:32.718350 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 605/1189] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:33:32 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:33:32.724229 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cee8a94-cbd8-4597-8a3d-2c155e93b8ff None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:32.822526 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cee8a94-cbd8-4597-8a3d-2c155e93b8ff None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:32.856151 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e844863a-14ee-4067-9155-81df1d4d7658 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:32.856787 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e844863a-14ee-4067-9155-81df1d4d7658 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:32.857067 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e844863a-14ee-4067-9155-81df1d4d7658 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:32.857347 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e844863a-14ee-4067-9155-81df1d4d7658 None admin] PATH_INFO: `/v3/users/9fc94d65946c4437aa9b1a9d770bd58c` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:32.858519 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e844863a-14ee-4067-9155-81df1d4d7658 None admin] RBAC: Authorizing `identity:get_user(user_id=9fc94d65946c4437aa9b1a9d770bd58c)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:33:32.859952 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e844863a-14ee-4067-9155-81df1d4d7658 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:33:32.861538 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 585/1190] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:33:32 2022] GET /identity/v3/users/9fc94d65946c4437aa9b1a9d770bd58c => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:33:32.867930 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0292bc69-53ea-4fc8-b9a2-b458c00daf91 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:32.928774 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:33:32.967573 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0292bc69-53ea-4fc8-b9a2-b458c00daf91 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:33.000312 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f6e0d7-1e19-475d-8d39-f4a1abe3d063 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:33.000928 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f6e0d7-1e19-475d-8d39-f4a1abe3d063 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:33.001207 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f6e0d7-1e19-475d-8d39-f4a1abe3d063 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:33.001481 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f6e0d7-1e19-475d-8d39-f4a1abe3d063 None admin] PATH_INFO: `/v3/projects/4fc6cd78acc4462f9fe5808abc6782e1` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:33.002593 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14f6e0d7-1e19-475d-8d39-f4a1abe3d063 None admin] RBAC: Authorizing `identity:get_project(project_id=4fc6cd78acc4462f9fe5808abc6782e1)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:33:33.003995 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14f6e0d7-1e19-475d-8d39-f4a1abe3d063 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:33:33.005674 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 606/1191] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:33:32 2022] GET /identity/v3/projects/4fc6cd78acc4462f9fe5808abc6782e1 => generated 361 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:33:33.011666 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e844863a-14ee-4067-9155-81df1d4d7658 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:33.043025 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:33:33.110597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e844863a-14ee-4067-9155-81df1d4d7658 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:33.149814 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7982905-0fc8-4251-9572-70e6a1f8c145 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:33.150535 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7982905-0fc8-4251-9572-70e6a1f8c145 None admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:33.150816 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7982905-0fc8-4251-9572-70e6a1f8c145 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:33.151091 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7982905-0fc8-4251-9572-70e6a1f8c145 None admin] PATH_INFO: `/v3/projects/4fc6cd78acc4462f9fe5808abc6782e1/users/9fc94d65946c4437aa9b1a9d770bd58c/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:33.153083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7982905-0fc8-4251-9572-70e6a1f8c145 None admin] RBAC: Authorizing `identity:create_grant(project_id=4fc6cd78acc4462f9fe5808abc6782e1, user_id=9fc94d65946c4437aa9b1a9d770bd58c, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:33:33.154579 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7982905-0fc8-4251-9572-70e6a1f8c145 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:33:33.169532 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 586/1192] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Oct 28 09:33:33 2022] PUT /identity/v3/projects/4fc6cd78acc4462f9fe5808abc6782e1/users/9fc94d65946c4437aa9b1a9d770bd58c/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:33:35.356718 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af1a7dc-11c6-4383-9d14-2a862c5def59 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:35.357420 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af1a7dc-11c6-4383-9d14-2a862c5def59 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:35.357420 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af1a7dc-11c6-4383-9d14-2a862c5def59 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:35.358649 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 607/1193] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:33:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 09:33:35.364233 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af377a9-a806-4097-8043-7b2d122b7d9f None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:35.364776 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af377a9-a806-4097-8043-7b2d122b7d9f None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:35.364916 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af377a9-a806-4097-8043-7b2d122b7d9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:35.385390 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-6af377a9-a806-4097-8043-7b2d122b7d9f None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:33:35.386641 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-6af377a9-a806-4097-8043-7b2d122b7d9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:33:35.464084 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 587/1194] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:33:35 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:33:35.734219 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af1a7dc-11c6-4383-9d14-2a862c5def59 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:35.841845 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af1a7dc-11c6-4383-9d14-2a862c5def59 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:35.872062 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:33:35.879463 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16fdfe63-229f-4717-a242-da1ece4c7ae0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:35.880117 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16fdfe63-229f-4717-a242-da1ece4c7ae0 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:35.880395 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16fdfe63-229f-4717-a242-da1ece4c7ae0 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:35.880667 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16fdfe63-229f-4717-a242-da1ece4c7ae0 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:35.891916 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16fdfe63-229f-4717-a242-da1ece4c7ae0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:33:35.897318 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-16fdfe63-229f-4717-a242-da1ece4c7ae0 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:33:35.897318 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:33:35.897318 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:33:35.897318 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:33:35.897318 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:33:35.897318 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:33:35.897318 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:33:35.897318 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:33:35.897318 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:33:35.897318 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:33:35.897318 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:33:35.897318 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:33:35.897318 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:33:35.897318 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:33:35.897318 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:33:35.897318 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:33:35.899116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:33:35.899116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:33:35.899116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:33:35.899116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:33:35.899116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:33:35.899116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:33:35.899116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:33:35.899116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:33:35.899116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:33:35.899116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:33:35.899116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:33:35.899116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:33:35.899116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:33:35.899116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:33:35.899116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:33:35.899116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:33:35.900727 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:33:35.900727 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:33:35.900727 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:33:35.900727 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:33:35.900727 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:33:35.900727 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:33:35.900727 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:33:35.900727 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:33:35.900727 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:33:35.900727 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:33:35.900727 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:33:35.900727 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:33:35.900727 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 608/1195] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Oct 28 09:33:35 2022] GET /identity/v3/roles/admin => generated 83 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:33:35.905305 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af377a9-a806-4097-8043-7b2d122b7d9f None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:35.934478 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:33:36.016843 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af377a9-a806-4097-8043-7b2d122b7d9f None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:36.052500 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36539615-1b63-4943-a2f9-d5fabe0945b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:36.053107 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36539615-1b63-4943-a2f9-d5fabe0945b5 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:36.053386 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36539615-1b63-4943-a2f9-d5fabe0945b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:36.053653 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36539615-1b63-4943-a2f9-d5fabe0945b5 None admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:36.054302 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36539615-1b63-4943-a2f9-d5fabe0945b5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:33:36.054734 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36539615-1b63-4943-a2f9-d5fabe0945b5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:33:36.056075 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36539615-1b63-4943-a2f9-d5fabe0945b5 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:33:36.064695 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 588/1196] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:33:35 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:33:36.073997 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16fdfe63-229f-4717-a242-da1ece4c7ae0 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:36.179603 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16fdfe63-229f-4717-a242-da1ece4c7ae0 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:36.216281 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cb7fb9b-629b-4fcc-8e9c-35025df51053 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:36.216913 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb7fb9b-629b-4fcc-8e9c-35025df51053 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:36.217196 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb7fb9b-629b-4fcc-8e9c-35025df51053 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:36.217471 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb7fb9b-629b-4fcc-8e9c-35025df51053 None admin] PATH_INFO: `/v3/users/9fc94d65946c4437aa9b1a9d770bd58c` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:36.218628 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cb7fb9b-629b-4fcc-8e9c-35025df51053 None admin] RBAC: Authorizing `identity:get_user(user_id=9fc94d65946c4437aa9b1a9d770bd58c)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:33:36.220048 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cb7fb9b-629b-4fcc-8e9c-35025df51053 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:33:36.221645 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 609/1197] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Oct 28 09:33:36 2022] GET /identity/v3/users/9fc94d65946c4437aa9b1a9d770bd58c => generated 311 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:33:36.228548 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36539615-1b63-4943-a2f9-d5fabe0945b5 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:36.313496 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:33:36.343803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36539615-1b63-4943-a2f9-d5fabe0945b5 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:36.401506 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d9abb60-f2e6-4a99-a162-d12160a30e34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:36.402562 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9abb60-f2e6-4a99-a162-d12160a30e34 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:36.403051 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9abb60-f2e6-4a99-a162-d12160a30e34 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:36.403522 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9abb60-f2e6-4a99-a162-d12160a30e34 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:36.517299 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d9abb60-f2e6-4a99-a162-d12160a30e34 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:33:36.520041 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d9abb60-f2e6-4a99-a162-d12160a30e34 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:33:36.655301 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:33:36.684888 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 589/1198] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:33:36 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 457 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 09:33:36.691236 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cb7fb9b-629b-4fcc-8e9c-35025df51053 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:36.753431 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:33:36.797000 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cb7fb9b-629b-4fcc-8e9c-35025df51053 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:36.832795 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d0d0de-c1d5-4820-bf0d-82a377e0f337 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:36.833973 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d0d0de-c1d5-4820-bf0d-82a377e0f337 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:36.834249 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d0d0de-c1d5-4820-bf0d-82a377e0f337 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:36.834555 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d0d0de-c1d5-4820-bf0d-82a377e0f337 None admin] PATH_INFO: `/v3/projects/4fc6cd78acc4462f9fe5808abc6782e1` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:36.835822 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d0d0de-c1d5-4820-bf0d-82a377e0f337 None admin] RBAC: Authorizing `identity:get_project(project_id=4fc6cd78acc4462f9fe5808abc6782e1)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:33:36.837297 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d0d0de-c1d5-4820-bf0d-82a377e0f337 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:33:36.839094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 610/1199] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Oct 28 09:33:36 2022] GET /identity/v3/projects/4fc6cd78acc4462f9fe5808abc6782e1 => generated 361 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:33:36.845777 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d9abb60-f2e6-4a99-a162-d12160a30e34 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:33:36.953531 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d9abb60-f2e6-4a99-a162-d12160a30e34 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:33:36.989118 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4446a530-401f-4547-90a8-ed12e1ac6c06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:33:36.989796 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4446a530-401f-4547-90a8-ed12e1ac6c06 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:33:36.990078 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4446a530-401f-4547-90a8-ed12e1ac6c06 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:33:36.990353 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4446a530-401f-4547-90a8-ed12e1ac6c06 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:33:36.991115 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4446a530-401f-4547-90a8-ed12e1ac6c06 None admin] RBAC: Adding query filter params (role.id=e13d798a98ea4c3dbb87a236ec683d32, scope.project.id=4fc6cd78acc4462f9fe5808abc6782e1, user.id=9fc94d65946c4437aa9b1a9d770bd58c) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:33:36.991432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4446a530-401f-4547-90a8-ed12e1ac6c06 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:33:36.992863 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4446a530-401f-4547-90a8-ed12e1ac6c06 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:33:36.999067 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 590/1200] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:33:36 2022] GET /identity/v3/role_assignments?user.id=9fc94d65946c4437aa9b1a9d770bd58c&scope.project.id=4fc6cd78acc4462f9fe5808abc6782e1&role.id=e13d798a98ea4c3dbb87a236ec683d32 => generated 608 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:33:46.828448 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=37 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:33:46.996283 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=77 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:34:37.775764 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42e1e08-6139-43d8-86c7-05c54993fc34 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:37.775764 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42e1e08-6139-43d8-86c7-05c54993fc34 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:37.775764 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42e1e08-6139-43d8-86c7-05c54993fc34 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:37.775764 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 611/1201] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:34:37 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:34:37.786514 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad653ed0-6d5b-4deb-b5cb-98d75b96af33 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:37.787986 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad653ed0-6d5b-4deb-b5cb-98d75b96af33 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:37.788722 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad653ed0-6d5b-4deb-b5cb-98d75b96af33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:37.812989 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-ad653ed0-6d5b-4deb-b5cb-98d75b96af33 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:34:37.814231 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-ad653ed0-6d5b-4deb-b5cb-98d75b96af33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:34:37.926735 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 591/1202] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:34:37 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:34:38.326745 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d42e1e08-6139-43d8-86c7-05c54993fc34 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:34:38.515566 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d42e1e08-6139-43d8-86c7-05c54993fc34 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:34:38.586677 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f9f1ff-9d2e-46bb-bd94-ebb17f6591df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:34:38.588220 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f9f1ff-9d2e-46bb-bd94-ebb17f6591df None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:38.588994 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f9f1ff-9d2e-46bb-bd94-ebb17f6591df None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:38.589743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f9f1ff-9d2e-46bb-bd94-ebb17f6591df None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:38.609531 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34f9f1ff-9d2e-46bb-bd94-ebb17f6591df None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:34:38.612827 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34f9f1ff-9d2e-46bb-bd94-ebb17f6591df None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:34:38.649238 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-34f9f1ff-9d2e-46bb-bd94-ebb17f6591df None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:34:38.649238 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:34:38.649238 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:34:38.649238 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:34:38.649238 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:34:38.649238 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:34:38.649238 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:34:38.649238 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:34:38.649238 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:34:38.649238 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:34:38.649238 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:34:38.649238 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:34:38.649238 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:34:38.649238 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:34:38.649238 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:34:38.649238 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:34:38.651525 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:34:38.651525 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:34:38.651525 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:34:38.651525 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:34:38.651525 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:34:38.651525 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:34:38.651525 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:34:38.651525 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:34:38.651525 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:34:38.651525 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:34:38.651525 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:34:38.651525 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:34:38.651525 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:34:38.651525 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:34:38.651525 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:34:38.651525 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:34:38.651525 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:34:38.653988 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:34:38.653988 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:34:38.653988 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:34:38.653988 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 612/1203] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:34:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 325 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:34:38.661701 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad653ed0-6d5b-4deb-b5cb-98d75b96af33 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:34:38.854153 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad653ed0-6d5b-4deb-b5cb-98d75b96af33 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:34:38.900355 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:34:38.911143 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9911fc4a-ca6a-4d70-9d72-6f9a8ca29706 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:34:38.911660 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9911fc4a-ca6a-4d70-9d72-6f9a8ca29706 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:38.911968 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9911fc4a-ca6a-4d70-9d72-6f9a8ca29706 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:38.912347 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9911fc4a-ca6a-4d70-9d72-6f9a8ca29706 None admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:38.913034 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9911fc4a-ca6a-4d70-9d72-6f9a8ca29706 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:34:38.913372 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9911fc4a-ca6a-4d70-9d72-6f9a8ca29706 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:34:38.915021 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9911fc4a-ca6a-4d70-9d72-6f9a8ca29706 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:34:38.937811 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 592/1204] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:34:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:34:38.951390 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f9f1ff-9d2e-46bb-bd94-ebb17f6591df None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:34:39.034180 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:34:39.127212 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f9f1ff-9d2e-46bb-bd94-ebb17f6591df None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:34:39.183148 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86d5f288-6b97-4fe3-b27e-119b5064badc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:34:39.184161 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d5f288-6b97-4fe3-b27e-119b5064badc None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:39.184626 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d5f288-6b97-4fe3-b27e-119b5064badc None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:39.185104 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86d5f288-6b97-4fe3-b27e-119b5064badc None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:39.186415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86d5f288-6b97-4fe3-b27e-119b5064badc None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:34:39.188919 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86d5f288-6b97-4fe3-b27e-119b5064badc None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:34:39.223717 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 613/1205] 10.222.0.25 () {60 vars in 1132 bytes} [Fri Oct 28 09:34:38 2022] POST /identity/v3/users => generated 251 bytes in 274 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:34:42.243071 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d002f8ea-6dbd-4bd0-a9b1-a58ba53e70df None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:42.243786 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d002f8ea-6dbd-4bd0-a9b1-a58ba53e70df None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:42.244959 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d002f8ea-6dbd-4bd0-a9b1-a58ba53e70df None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:42.248430 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 593/1206] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:34:42 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:34:42.260695 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb79dcd9-00ec-4333-9d14-6e03a19b529e None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:42.261284 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb79dcd9-00ec-4333-9d14-6e03a19b529e None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:42.261698 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb79dcd9-00ec-4333-9d14-6e03a19b529e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:42.297382 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-eb79dcd9-00ec-4333-9d14-6e03a19b529e None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:34:42.299454 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-eb79dcd9-00ec-4333-9d14-6e03a19b529e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:34:42.438870 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 614/1207] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:34:42 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 09:34:42.846689 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d002f8ea-6dbd-4bd0-a9b1-a58ba53e70df None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:34:43.027348 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d002f8ea-6dbd-4bd0-a9b1-a58ba53e70df None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:34:43.089601 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac29ec45-c789-4f68-9986-73d827ff5e89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:34:43.090658 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac29ec45-c789-4f68-9986-73d827ff5e89 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:43.091072 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac29ec45-c789-4f68-9986-73d827ff5e89 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:43.091612 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac29ec45-c789-4f68-9986-73d827ff5e89 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:43.109123 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac29ec45-c789-4f68-9986-73d827ff5e89 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:34:43.117103 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-ac29ec45-c789-4f68-9986-73d827ff5e89 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 09:34:43.117103 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:34:43.117103 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:34:43.117103 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:34:43.117103 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:34:43.117103 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:34:43.117103 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:34:43.117103 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:34:43.117103 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:34:43.117103 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:34:43.117103 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:34:43.117103 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:34:43.117103 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:34:43.117103 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:34:43.117103 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:34:43.117103 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:34:43.120873 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:34:43.120873 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:34:43.120873 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:34:43.120873 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:34:43.120873 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:34:43.120873 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:34:43.120873 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:34:43.120873 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:34:43.120873 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:34:43.120873 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:34:43.120873 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:34:43.120873 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:34:43.120873 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:34:43.120873 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:34:43.120873 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:34:43.120873 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:34:43.123031 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:34:43.123031 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:34:43.123031 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:34:43.123031 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:34:43.123031 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:34:43.123031 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:34:43.123031 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:34:43.123031 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:34:43.123031 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:34:43.123031 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:34:43.123031 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 09:34:43.123031 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:34:43.123031 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 594/1208] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Oct 28 09:34:42 2022] GET /identity/v3/roles/service => generated 85 bytes in 273 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:34:43.129604 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb79dcd9-00ec-4333-9d14-6e03a19b529e None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:34:43.204816 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:34:43.336241 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb79dcd9-00ec-4333-9d14-6e03a19b529e None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:34:43.411826 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c8e95ea-8270-4420-8ffd-54723329da51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:34:43.412817 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8e95ea-8270-4420-8ffd-54723329da51 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:43.413380 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8e95ea-8270-4420-8ffd-54723329da51 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:43.413847 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8e95ea-8270-4420-8ffd-54723329da51 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:43.415000 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c8e95ea-8270-4420-8ffd-54723329da51 None admin] RBAC: Adding query filter params (name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:34:43.416061 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c8e95ea-8270-4420-8ffd-54723329da51 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:34:43.417572 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c8e95ea-8270-4420-8ffd-54723329da51 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:34:43.435858 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 615/1209] 10.222.0.25 () {56 vars in 1103 bytes} [Fri Oct 28 09:34:43 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 308 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:34:43.447843 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac29ec45-c789-4f68-9986-73d827ff5e89 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:34:43.542852 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:34:43.619904 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac29ec45-c789-4f68-9986-73d827ff5e89 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:34:43.666170 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c468de-7795-452b-b17a-a23cb261b04e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:34:43.666798 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c468de-7795-452b-b17a-a23cb261b04e None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:43.667114 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c468de-7795-452b-b17a-a23cb261b04e None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:43.667392 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c468de-7795-452b-b17a-a23cb261b04e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:43.680694 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c468de-7795-452b-b17a-a23cb261b04e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:34:43.682253 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c468de-7795-452b-b17a-a23cb261b04e None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:34:43.700597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-c2c468de-7795-452b-b17a-a23cb261b04e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:34:43.700597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:34:43.700597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:34:43.700597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:34:43.700597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:34:43.700597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:34:43.700597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:34:43.700597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:34:43.700597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:34:43.700597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:34:43.700597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:34:43.700597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:34:43.700597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:34:43.700597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:34:43.700597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:34:43.700597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:34:43.702445 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:34:43.702445 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:34:43.702445 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:34:43.702445 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:34:43.702445 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:34:43.702445 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:34:43.702445 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:34:43.702445 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:34:43.702445 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:34:43.702445 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:34:43.702445 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:34:43.702445 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:34:43.702445 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:34:43.702445 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:34:43.702445 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:34:43.702445 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:34:43.702445 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:34:43.703800 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:34:43.703800 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:34:43.703800 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:34:43.703800 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 595/1210] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:34:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 256 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:34:43.712537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c8e95ea-8270-4420-8ffd-54723329da51 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:34:43.887807 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:34:43.895175 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c8e95ea-8270-4420-8ffd-54723329da51 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:34:43.949620 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81deef4-8b2a-457a-98da-2b9a58e5d21f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:34:43.950301 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81deef4-8b2a-457a-98da-2b9a58e5d21f None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:43.950731 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81deef4-8b2a-457a-98da-2b9a58e5d21f None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:43.951156 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81deef4-8b2a-457a-98da-2b9a58e5d21f None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:43.952030 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b81deef4-8b2a-457a-98da-2b9a58e5d21f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:34:43.952477 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b81deef4-8b2a-457a-98da-2b9a58e5d21f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:34:43.954000 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b81deef4-8b2a-457a-98da-2b9a58e5d21f None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:34:43.988845 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 616/1211] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:34:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:34:43.991244 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c468de-7795-452b-b17a-a23cb261b04e None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:34:44.189640 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c468de-7795-452b-b17a-a23cb261b04e None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:34:44.192928 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:34:44.263647 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b102b95c-40e1-4bdf-b211-f44dd4662f47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:34:44.265247 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b102b95c-40e1-4bdf-b211-f44dd4662f47 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:44.265915 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b102b95c-40e1-4bdf-b211-f44dd4662f47 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:44.266666 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b102b95c-40e1-4bdf-b211-f44dd4662f47 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:44.283022 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b102b95c-40e1-4bdf-b211-f44dd4662f47 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:34:44.286966 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b102b95c-40e1-4bdf-b211-f44dd4662f47 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:34:44.314571 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-b102b95c-40e1-4bdf-b211-f44dd4662f47 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 28 09:34:44.314571 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:34:44.314571 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:34:44.314571 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:34:44.314571 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:34:44.314571 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:34:44.314571 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:34:44.314571 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:34:44.314571 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:34:44.314571 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:34:44.314571 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:34:44.314571 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:34:44.314571 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 09:34:44.314571 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 09:34:44.314571 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 09:34:44.314571 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 09:34:44.316581 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:34:44.316581 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:34:44.316581 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 09:34:44.316581 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:34:44.316581 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 09:34:44.316581 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:34:44.316581 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:34:44.316581 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:34:44.316581 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:34:44.316581 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:34:44.316581 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:34:44.316581 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:34:44.316581 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:34:44.316581 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:34:44.316581 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:34:44.316581 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:34:44.316581 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:34:44.316581 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:34:44.319319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:34:44.319319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:34:44.319319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 09:34:44.319319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 09:34:44.319319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 09:34:44.319319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 09:34:44.319319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 09:34:44.319319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 09:34:44.319319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 28 09:34:44.319319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:34:44.319319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 596/1212] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:34:43 2022] GET /identity/v3/users/placement => generated 87 bytes in 327 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:34:44.326184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81deef4-8b2a-457a-98da-2b9a58e5d21f None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:34:44.499725 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81deef4-8b2a-457a-98da-2b9a58e5d21f None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:34:44.568049 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22bf18e2-958f-4048-bbb7-97548c76c5b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:34:44.568991 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22bf18e2-958f-4048-bbb7-97548c76c5b5 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:44.569466 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22bf18e2-958f-4048-bbb7-97548c76c5b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:44.569955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22bf18e2-958f-4048-bbb7-97548c76c5b5 None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:44.571367 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22bf18e2-958f-4048-bbb7-97548c76c5b5 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:34:44.571907 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22bf18e2-958f-4048-bbb7-97548c76c5b5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:34:44.574526 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22bf18e2-958f-4048-bbb7-97548c76c5b5 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:34:44.603721 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 617/1213] 10.222.0.25 () {56 vars in 1143 bytes} [Fri Oct 28 09:34:44 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:34:44.608890 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b102b95c-40e1-4bdf-b211-f44dd4662f47 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:34:44.734085 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b102b95c-40e1-4bdf-b211-f44dd4662f47 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:34:44.785077 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9419f4c7-2b0f-4856-8eb8-78693ff9f58a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:34:44.786001 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9419f4c7-2b0f-4856-8eb8-78693ff9f58a None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:44.786314 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9419f4c7-2b0f-4856-8eb8-78693ff9f58a None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:44.786660 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9419f4c7-2b0f-4856-8eb8-78693ff9f58a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:44.824413 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:34:44.880979 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9419f4c7-2b0f-4856-8eb8-78693ff9f58a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:34:44.882610 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9419f4c7-2b0f-4856-8eb8-78693ff9f58a None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:34:45.026212 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 597/1214] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:34:44 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 418 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 09:34:45.037629 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22bf18e2-958f-4048-bbb7-97548c76c5b5 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:34:45.063291 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:34:45.224471 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22bf18e2-958f-4048-bbb7-97548c76c5b5 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:34:45.268409 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11d7c31a-57fc-4a61-b040-fb41c8866721 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:34:45.269136 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d7c31a-57fc-4a61-b040-fb41c8866721 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:45.269359 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d7c31a-57fc-4a61-b040-fb41c8866721 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:45.269716 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d7c31a-57fc-4a61-b040-fb41c8866721 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:45.281428 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11d7c31a-57fc-4a61-b040-fb41c8866721 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:34:45.283142 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11d7c31a-57fc-4a61-b040-fb41c8866721 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:34:45.302293 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-11d7c31a-57fc-4a61-b040-fb41c8866721 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:34:45.302293 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:34:45.302293 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:34:45.302293 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:34:45.302293 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:34:45.302293 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:34:45.302293 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:34:45.302293 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:34:45.302293 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:34:45.302293 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:34:45.302293 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:34:45.302293 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:34:45.302293 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:34:45.302293 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:34:45.302293 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:34:45.302293 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:34:45.312592 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:34:45.312592 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:34:45.312592 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:34:45.312592 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:34:45.312592 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:34:45.312592 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:34:45.312592 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:34:45.312592 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:34:45.312592 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:34:45.312592 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:34:45.312592 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:34:45.312592 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:34:45.312592 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:34:45.312592 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:34:45.312592 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:34:45.312592 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:34:45.312592 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:34:45.314605 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:34:45.314605 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:34:45.314605 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:34:45.314605 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 618/1215] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:34:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 267 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:34:45.315117 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9419f4c7-2b0f-4856-8eb8-78693ff9f58a None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:34:45.434355 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:34:45.496679 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9419f4c7-2b0f-4856-8eb8-78693ff9f58a None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:34:45.553892 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f372a4ac-c7a8-4d83-9ed9-dbdcc87f677c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:34:45.554742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f372a4ac-c7a8-4d83-9ed9-dbdcc87f677c None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:45.555237 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f372a4ac-c7a8-4d83-9ed9-dbdcc87f677c None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:45.555668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f372a4ac-c7a8-4d83-9ed9-dbdcc87f677c None admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:45.556755 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f372a4ac-c7a8-4d83-9ed9-dbdcc87f677c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:34:45.557209 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f372a4ac-c7a8-4d83-9ed9-dbdcc87f677c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:34:45.559721 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f372a4ac-c7a8-4d83-9ed9-dbdcc87f677c None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:34:45.583583 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 598/1216] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:34:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:34:45.593446 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11d7c31a-57fc-4a61-b040-fb41c8866721 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:34:45.709414 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:34:45.754317 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11d7c31a-57fc-4a61-b040-fb41c8866721 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:34:45.798730 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b642865d-55ff-4d0e-9f9a-e07b86d887d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:34:45.799469 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b642865d-55ff-4d0e-9f9a-e07b86d887d1 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:45.799738 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b642865d-55ff-4d0e-9f9a-e07b86d887d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:45.800017 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b642865d-55ff-4d0e-9f9a-e07b86d887d1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:45.808929 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b642865d-55ff-4d0e-9f9a-e07b86d887d1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:34:45.810415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b642865d-55ff-4d0e-9f9a-e07b86d887d1 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:34:45.825463 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-b642865d-55ff-4d0e-9f9a-e07b86d887d1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:34:45.825463 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:34:45.825463 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:34:45.825463 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:34:45.825463 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:34:45.825463 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:34:45.825463 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:34:45.825463 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:34:45.825463 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:34:45.825463 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:34:45.825463 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:34:45.825463 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:34:45.825463 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 09:34:45.825463 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 09:34:45.825463 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 09:34:45.826541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 09:34:45.826541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:34:45.826541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:34:45.826541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:34:45.826541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:34:45.826541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:34:45.826541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:34:45.826541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:34:45.826541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:34:45.826541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:34:45.826541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:34:45.826541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:34:45.826541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:34:45.826541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:34:45.826541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:34:45.826541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:34:45.826541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:34:45.827536 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 09:34:45.827536 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 09:34:45.827536 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 09:34:45.827536 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 09:34:45.827536 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 09:34:45.827536 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 09:34:45.827536 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:34:45.827536 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:34:45.827536 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 619/1217] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:34:45 2022] GET /identity/v3/projects/service => generated 88 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:34:45.833777 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f372a4ac-c7a8-4d83-9ed9-dbdcc87f677c None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:34:45.988495 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f372a4ac-c7a8-4d83-9ed9-dbdcc87f677c None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:34:46.025533 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:34:46.040593 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f6a0c0-e2f5-432a-b2d5-3975be5daca5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:34:46.041284 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f6a0c0-e2f5-432a-b2d5-3975be5daca5 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:46.041605 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f6a0c0-e2f5-432a-b2d5-3975be5daca5 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:46.041892 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f6a0c0-e2f5-432a-b2d5-3975be5daca5 None admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:46.042794 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f6a0c0-e2f5-432a-b2d5-3975be5daca5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:34:46.043105 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f6a0c0-e2f5-432a-b2d5-3975be5daca5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:34:46.044570 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f6a0c0-e2f5-432a-b2d5-3975be5daca5 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:34:46.060448 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 599/1218] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Oct 28 09:34:45 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:34:46.068204 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b642865d-55ff-4d0e-9f9a-e07b86d887d1 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:34:46.234869 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b642865d-55ff-4d0e-9f9a-e07b86d887d1 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:34:46.272091 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:34:46.280867 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e0ef9e-6ed5-4263-bd61-0b8687d7425b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:34:46.281751 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e0ef9e-6ed5-4263-bd61-0b8687d7425b None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:46.282218 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e0ef9e-6ed5-4263-bd61-0b8687d7425b None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:46.282777 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e0ef9e-6ed5-4263-bd61-0b8687d7425b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:46.283726 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e0ef9e-6ed5-4263-bd61-0b8687d7425b None admin] RBAC: Adding query filter params (role.id=e5d8d090857a4a018d0ec3c79ece6472, scope.project.id=455d1c826b974bc38497d18a516867c7, user.id=3f1d95049d2d4593bf52670b1a0035fa) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:34:46.284230 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e0ef9e-6ed5-4263-bd61-0b8687d7425b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:34:46.286425 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e0ef9e-6ed5-4263-bd61-0b8687d7425b None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:34:46.296908 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 620/1219] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:34:46 2022] GET /identity/v3/role_assignments?user.id=3f1d95049d2d4593bf52670b1a0035fa&scope.project.id=455d1c826b974bc38497d18a516867c7&role.id=e5d8d090857a4a018d0ec3c79ece6472 => generated 259 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:34:48.680109 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91afb04-98bc-45bb-8665-42f08df47f68 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:48.680701 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91afb04-98bc-45bb-8665-42f08df47f68 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:48.680701 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91afb04-98bc-45bb-8665-42f08df47f68 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:48.682042 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 600/1220] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:34:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:34:48.688973 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112d1fd8-61b9-4c6d-bb02-868dd4aa3dca None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:48.689521 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112d1fd8-61b9-4c6d-bb02-868dd4aa3dca None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:48.689778 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112d1fd8-61b9-4c6d-bb02-868dd4aa3dca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:48.712264 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-112d1fd8-61b9-4c6d-bb02-868dd4aa3dca None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:34:48.713540 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-112d1fd8-61b9-4c6d-bb02-868dd4aa3dca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:34:48.790774 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 621/1221] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:34:48 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:34:49.074939 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f91afb04-98bc-45bb-8665-42f08df47f68 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:34:49.202278 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f91afb04-98bc-45bb-8665-42f08df47f68 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:34:49.249433 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c492bb-dddd-4129-a0fe-455701ee4dff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:34:49.249785 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c492bb-dddd-4129-a0fe-455701ee4dff None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:49.249921 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c492bb-dddd-4129-a0fe-455701ee4dff None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:49.250205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c492bb-dddd-4129-a0fe-455701ee4dff None admin] PATH_INFO: `/v3/roles/service` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:49.264354 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5c492bb-dddd-4129-a0fe-455701ee4dff None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:34:49.269922 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-e5c492bb-dddd-4129-a0fe-455701ee4dff None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 09:34:49.269922 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:34:49.269922 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:34:49.269922 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:34:49.269922 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:34:49.269922 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:34:49.269922 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:34:49.269922 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:34:49.269922 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:34:49.269922 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:34:49.269922 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:34:49.269922 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:34:49.269922 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:34:49.269922 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:34:49.269922 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:34:49.269922 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:34:49.270943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:34:49.270943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:34:49.270943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:34:49.270943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:34:49.270943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:34:49.270943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:34:49.270943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:34:49.270943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:34:49.270943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:34:49.270943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:34:49.270943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:34:49.270943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:34:49.270943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:34:49.270943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:34:49.270943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:34:49.270943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:34:49.271831 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:34:49.271831 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:34:49.271831 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:34:49.271831 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:34:49.271831 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:34:49.271831 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:34:49.271831 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:34:49.271831 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:34:49.271831 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:34:49.271831 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:34:49.271831 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 09:34:49.271831 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:34:49.274607 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 601/1222] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Oct 28 09:34:49 2022] GET /identity/v3/roles/service => generated 85 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:34:49.277620 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112d1fd8-61b9-4c6d-bb02-868dd4aa3dca None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:34:49.412187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112d1fd8-61b9-4c6d-bb02-868dd4aa3dca None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:34:49.443736 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:34:49.464580 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0cfa49c-9f1a-4df2-bbfc-7b09edba3209 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:34:49.465383 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cfa49c-9f1a-4df2-bbfc-7b09edba3209 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:49.465864 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cfa49c-9f1a-4df2-bbfc-7b09edba3209 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:49.466335 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cfa49c-9f1a-4df2-bbfc-7b09edba3209 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:49.467591 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0cfa49c-9f1a-4df2-bbfc-7b09edba3209 None admin] RBAC: Adding query filter params (name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:34:49.468131 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0cfa49c-9f1a-4df2-bbfc-7b09edba3209 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:34:49.470733 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0cfa49c-9f1a-4df2-bbfc-7b09edba3209 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:34:49.486824 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 622/1223] 10.222.0.25 () {56 vars in 1103 bytes} [Fri Oct 28 09:34:49 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:34:49.493207 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c492bb-dddd-4129-a0fe-455701ee4dff None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:34:49.557144 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:34:49.638295 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c492bb-dddd-4129-a0fe-455701ee4dff None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:34:49.700910 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c01bd78-ad5f-4c1c-90d6-3851b0316e6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:34:49.701883 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c01bd78-ad5f-4c1c-90d6-3851b0316e6f None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:49.702348 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c01bd78-ad5f-4c1c-90d6-3851b0316e6f None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:49.702856 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c01bd78-ad5f-4c1c-90d6-3851b0316e6f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:49.719634 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c01bd78-ad5f-4c1c-90d6-3851b0316e6f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:34:49.721143 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c01bd78-ad5f-4c1c-90d6-3851b0316e6f None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:34:49.743781 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-0c01bd78-ad5f-4c1c-90d6-3851b0316e6f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:34:49.743781 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:34:49.743781 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:34:49.743781 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:34:49.743781 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:34:49.743781 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:34:49.743781 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:34:49.743781 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:34:49.743781 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:34:49.743781 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:34:49.743781 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:34:49.743781 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:34:49.743781 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:34:49.743781 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:34:49.743781 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:34:49.743781 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:34:49.745895 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:34:49.745895 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:34:49.745895 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:34:49.745895 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:34:49.745895 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:34:49.745895 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:34:49.745895 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:34:49.745895 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:34:49.745895 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:34:49.745895 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:34:49.745895 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:34:49.745895 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:34:49.745895 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:34:49.745895 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:34:49.745895 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:34:49.745895 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:34:49.745895 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:34:49.747534 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:34:49.747534 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:34:49.747534 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:34:49.747534 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 602/1224] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:34:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:34:49.752744 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0cfa49c-9f1a-4df2-bbfc-7b09edba3209 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:34:49.905658 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0cfa49c-9f1a-4df2-bbfc-7b09edba3209 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:34:49.960229 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3798968-6557-47a7-a79b-34b8596e048a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:34:49.960958 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3798968-6557-47a7-a79b-34b8596e048a None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:49.961396 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3798968-6557-47a7-a79b-34b8596e048a None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:49.961783 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3798968-6557-47a7-a79b-34b8596e048a None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:49.962759 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3798968-6557-47a7-a79b-34b8596e048a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:34:49.963314 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3798968-6557-47a7-a79b-34b8596e048a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:34:49.965810 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3798968-6557-47a7-a79b-34b8596e048a None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:34:49.994803 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 623/1225] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:34:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:34:49.996928 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c01bd78-ad5f-4c1c-90d6-3851b0316e6f None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:34:50.135536 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:34:50.148438 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c01bd78-ad5f-4c1c-90d6-3851b0316e6f None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:34:50.210369 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53173c69-5638-453f-9cb8-b28ffcbb7485 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:34:50.211091 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53173c69-5638-453f-9cb8-b28ffcbb7485 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:50.211420 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53173c69-5638-453f-9cb8-b28ffcbb7485 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:50.211897 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53173c69-5638-453f-9cb8-b28ffcbb7485 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:50.223080 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53173c69-5638-453f-9cb8-b28ffcbb7485 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:34:50.225053 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53173c69-5638-453f-9cb8-b28ffcbb7485 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:34:50.237859 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-53173c69-5638-453f-9cb8-b28ffcbb7485 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 28 09:34:50.237859 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:34:50.237859 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:34:50.237859 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:34:50.237859 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:34:50.237859 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:34:50.237859 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:34:50.237859 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:34:50.237859 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:34:50.237859 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:34:50.237859 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:34:50.237859 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:34:50.237859 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 09:34:50.237859 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 09:34:50.237859 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 09:34:50.237859 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 09:34:50.240374 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:34:50.240374 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:34:50.240374 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 09:34:50.240374 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:34:50.240374 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 09:34:50.240374 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:34:50.240374 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:34:50.240374 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:34:50.240374 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:34:50.240374 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:34:50.240374 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:34:50.240374 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:34:50.240374 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:34:50.240374 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:34:50.240374 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:34:50.240374 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:34:50.240374 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:34:50.240374 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:34:50.242567 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:34:50.242567 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:34:50.242567 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 09:34:50.242567 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 09:34:50.242567 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 09:34:50.242567 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 09:34:50.242567 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 09:34:50.242567 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 09:34:50.242567 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 28 09:34:50.242567 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:34:50.242567 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 603/1226] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:34:49 2022] GET /identity/v3/users/placement => generated 87 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:34:50.245800 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3798968-6557-47a7-a79b-34b8596e048a None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:34:50.310428 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:34:50.392792 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3798968-6557-47a7-a79b-34b8596e048a None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:34:50.446585 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f61935e9-fbae-4187-a3ae-3187d9f29585 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:34:50.447571 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61935e9-fbae-4187-a3ae-3187d9f29585 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:50.447934 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61935e9-fbae-4187-a3ae-3187d9f29585 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:50.448287 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61935e9-fbae-4187-a3ae-3187d9f29585 None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:50.449451 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f61935e9-fbae-4187-a3ae-3187d9f29585 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:34:50.449917 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f61935e9-fbae-4187-a3ae-3187d9f29585 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:34:50.452009 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f61935e9-fbae-4187-a3ae-3187d9f29585 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:34:50.479424 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 624/1227] 10.222.0.25 () {56 vars in 1143 bytes} [Fri Oct 28 09:34:50 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:34:50.486985 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53173c69-5638-453f-9cb8-b28ffcbb7485 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:34:50.646320 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53173c69-5638-453f-9cb8-b28ffcbb7485 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:34:50.686966 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e083bb84-1284-4792-984c-a0b2b1439419 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:34:50.687940 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e083bb84-1284-4792-984c-a0b2b1439419 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:50.688411 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e083bb84-1284-4792-984c-a0b2b1439419 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:50.688910 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e083bb84-1284-4792-984c-a0b2b1439419 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:50.705701 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e083bb84-1284-4792-984c-a0b2b1439419 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:34:50.707619 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e083bb84-1284-4792-984c-a0b2b1439419 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:34:50.724165 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-e083bb84-1284-4792-984c-a0b2b1439419 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:34:50.724165 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:34:50.724165 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:34:50.724165 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:34:50.724165 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:34:50.724165 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:34:50.724165 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:34:50.724165 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:34:50.724165 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:34:50.724165 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:34:50.724165 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:34:50.724165 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:34:50.724165 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:34:50.724165 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:34:50.724165 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:34:50.724165 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:34:50.726230 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:34:50.726230 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:34:50.726230 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:34:50.726230 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:34:50.726230 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:34:50.726230 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:34:50.726230 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:34:50.726230 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:34:50.726230 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:34:50.726230 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:34:50.726230 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:34:50.726230 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:34:50.726230 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:34:50.726230 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:34:50.726230 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:34:50.726230 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:34:50.726230 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:34:50.728322 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:34:50.728322 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:34:50.728322 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:34:50.728322 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 604/1228] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:34:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:34:50.734487 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f61935e9-fbae-4187-a3ae-3187d9f29585 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:34:50.856074 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f61935e9-fbae-4187-a3ae-3187d9f29585 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:34:50.858709 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:34:50.908714 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-278c151a-168d-4cef-9ad4-03a61b660451 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:34:50.909710 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-278c151a-168d-4cef-9ad4-03a61b660451 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:50.910252 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-278c151a-168d-4cef-9ad4-03a61b660451 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:50.911034 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-278c151a-168d-4cef-9ad4-03a61b660451 None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:50.912163 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-278c151a-168d-4cef-9ad4-03a61b660451 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:34:50.912693 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-278c151a-168d-4cef-9ad4-03a61b660451 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:34:50.915181 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-278c151a-168d-4cef-9ad4-03a61b660451 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:34:50.936421 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 625/1229] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:34:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:34:50.937800 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e083bb84-1284-4792-984c-a0b2b1439419 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:34:50.947918 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:34:51.092397 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e083bb84-1284-4792-984c-a0b2b1439419 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:34:51.138955 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc343fc-c659-4787-a83c-a078dc9415f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:34:51.140033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc343fc-c659-4787-a83c-a078dc9415f7 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:51.140285 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc343fc-c659-4787-a83c-a078dc9415f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:51.140716 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edc343fc-c659-4787-a83c-a078dc9415f7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:51.149303 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edc343fc-c659-4787-a83c-a078dc9415f7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:34:51.150952 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edc343fc-c659-4787-a83c-a078dc9415f7 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:34:51.164974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-edc343fc-c659-4787-a83c-a078dc9415f7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:34:51.164974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:34:51.164974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:34:51.164974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:34:51.164974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:34:51.164974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:34:51.164974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:34:51.164974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:34:51.164974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:34:51.164974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:34:51.164974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:34:51.164974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:34:51.164974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 09:34:51.164974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 09:34:51.164974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 09:34:51.166128 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 09:34:51.166128 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:34:51.166128 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:34:51.166128 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:34:51.166128 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:34:51.166128 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:34:51.166128 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:34:51.166128 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:34:51.166128 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:34:51.166128 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:34:51.166128 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:34:51.166128 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:34:51.166128 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:34:51.166128 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:34:51.166128 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:34:51.166128 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:34:51.166128 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:34:51.167220 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 09:34:51.167220 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 09:34:51.167220 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 09:34:51.167220 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 09:34:51.167220 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 09:34:51.167220 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 09:34:51.167220 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:34:51.167220 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:34:51.171476 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 605/1230] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:34:50 2022] GET /identity/v3/projects/service => generated 88 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:34:51.173199 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-278c151a-168d-4cef-9ad4-03a61b660451 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:34:51.286655 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-278c151a-168d-4cef-9ad4-03a61b660451 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:34:51.324372 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff4959f-d063-42aa-a9fb-fc4a38d8c469 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:34:51.324993 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff4959f-d063-42aa-a9fb-fc4a38d8c469 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:51.325266 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff4959f-d063-42aa-a9fb-fc4a38d8c469 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:51.325766 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff4959f-d063-42aa-a9fb-fc4a38d8c469 None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:51.326608 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff4959f-d063-42aa-a9fb-fc4a38d8c469 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:34:51.327113 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff4959f-d063-42aa-a9fb-fc4a38d8c469 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:34:51.328630 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff4959f-d063-42aa-a9fb-fc4a38d8c469 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:34:51.344534 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 626/1231] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Oct 28 09:34:51 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:34:51.352100 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc343fc-c659-4787-a83c-a078dc9415f7 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:34:51.448528 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:34:51.486982 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edc343fc-c659-4787-a83c-a078dc9415f7 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:34:51.525936 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090104ab-6f7b-4fc2-8aa1-c123051ed091 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:34:51.526706 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090104ab-6f7b-4fc2-8aa1-c123051ed091 None admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:51.526998 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090104ab-6f7b-4fc2-8aa1-c123051ed091 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:51.527274 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090104ab-6f7b-4fc2-8aa1-c123051ed091 None admin] PATH_INFO: `/v3/projects/455d1c826b974bc38497d18a516867c7/users/3f1d95049d2d4593bf52670b1a0035fa/roles/e5d8d090857a4a018d0ec3c79ece6472` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:51.544083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-090104ab-6f7b-4fc2-8aa1-c123051ed091 None admin] RBAC: Authorizing `identity:create_grant(project_id=455d1c826b974bc38497d18a516867c7, user_id=3f1d95049d2d4593bf52670b1a0035fa, role_id=e5d8d090857a4a018d0ec3c79ece6472)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:34:51.545542 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-090104ab-6f7b-4fc2-8aa1-c123051ed091 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:34:51.562360 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 606/1232] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Oct 28 09:34:51 2022] PUT /identity/v3/projects/455d1c826b974bc38497d18a516867c7/users/3f1d95049d2d4593bf52670b1a0035fa/roles/e5d8d090857a4a018d0ec3c79ece6472 => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:34:54.030749 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13891b0-acd7-40e4-a4e9-322c97d027a2 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:54.031323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13891b0-acd7-40e4-a4e9-322c97d027a2 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:54.031721 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13891b0-acd7-40e4-a4e9-322c97d027a2 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:54.034364 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 627/1233] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:34:54 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:34:54.041220 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3bd6ddd-19bd-4903-8dcf-2335f87eb68f None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:54.042018 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3bd6ddd-19bd-4903-8dcf-2335f87eb68f None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:54.042018 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3bd6ddd-19bd-4903-8dcf-2335f87eb68f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:54.063245 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-f3bd6ddd-19bd-4903-8dcf-2335f87eb68f None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:34:54.064493 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-f3bd6ddd-19bd-4903-8dcf-2335f87eb68f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:34:54.159016 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 607/1234] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:34:54 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:34:54.444217 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13891b0-acd7-40e4-a4e9-322c97d027a2 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:34:54.481931 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:34:54.609696 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13891b0-acd7-40e4-a4e9-322c97d027a2 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:34:54.651507 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5601ed59-9257-4d39-9962-2c2d94e68813 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:34:54.652251 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5601ed59-9257-4d39-9962-2c2d94e68813 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:54.652673 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5601ed59-9257-4d39-9962-2c2d94e68813 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:54.653138 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5601ed59-9257-4d39-9962-2c2d94e68813 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:54.667268 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5601ed59-9257-4d39-9962-2c2d94e68813 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:34:54.672604 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-5601ed59-9257-4d39-9962-2c2d94e68813 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Oct 28 09:34:54.672604 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:34:54.672604 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:34:54.672604 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:34:54.672604 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:34:54.672604 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:34:54.672604 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:34:54.672604 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:34:54.672604 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:34:54.672604 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:34:54.672604 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:34:54.672604 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:34:54.672604 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:34:54.672604 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:34:54.672604 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:34:54.672604 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:34:54.674723 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:34:54.674723 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:34:54.674723 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:34:54.674723 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:34:54.674723 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:34:54.674723 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:34:54.674723 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:34:54.674723 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:34:54.674723 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:34:54.674723 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:34:54.674723 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:34:54.674723 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:34:54.674723 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:34:54.674723 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:34:54.674723 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:34:54.674723 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:34:54.676952 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:34:54.676952 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:34:54.676952 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:34:54.676952 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:34:54.676952 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:34:54.676952 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:34:54.676952 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:34:54.676952 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:34:54.676952 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:34:54.676952 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:34:54.676952 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Oct 28 09:34:54.676952 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:34:54.676952 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 628/1235] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Oct 28 09:34:54 2022] GET /identity/v3/roles/service => generated 85 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:34:54.683895 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3bd6ddd-19bd-4903-8dcf-2335f87eb68f None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:34:54.763039 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:34:54.857358 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3bd6ddd-19bd-4903-8dcf-2335f87eb68f None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:34:54.906765 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae73033-feb5-4a3e-a43d-406748becd73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:34:54.907896 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae73033-feb5-4a3e-a43d-406748becd73 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:54.908453 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae73033-feb5-4a3e-a43d-406748becd73 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:54.908928 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae73033-feb5-4a3e-a43d-406748becd73 None admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:54.911104 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bae73033-feb5-4a3e-a43d-406748becd73 None admin] RBAC: Adding query filter params (name=service) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:34:54.911655 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bae73033-feb5-4a3e-a43d-406748becd73 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:34:54.914669 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bae73033-feb5-4a3e-a43d-406748becd73 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:34:54.936772 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 608/1236] 10.222.0.25 () {56 vars in 1103 bytes} [Fri Oct 28 09:34:54 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:34:54.938689 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5601ed59-9257-4d39-9962-2c2d94e68813 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:34:55.090100 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:34:55.119988 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5601ed59-9257-4d39-9962-2c2d94e68813 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:34:55.179529 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d7a4791-8fd0-47c0-b653-31c0fc091331 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:34:55.180246 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7a4791-8fd0-47c0-b653-31c0fc091331 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:55.180529 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7a4791-8fd0-47c0-b653-31c0fc091331 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:55.180881 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7a4791-8fd0-47c0-b653-31c0fc091331 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:55.200453 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d7a4791-8fd0-47c0-b653-31c0fc091331 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:34:55.203535 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d7a4791-8fd0-47c0-b653-31c0fc091331 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:34:55.234940 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-6d7a4791-8fd0-47c0-b653-31c0fc091331 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:34:55.234940 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:34:55.234940 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:34:55.234940 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:34:55.234940 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:34:55.234940 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:34:55.234940 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:34:55.234940 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:34:55.234940 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:34:55.234940 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:34:55.234940 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:34:55.234940 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:34:55.234940 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:34:55.234940 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:34:55.234940 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:34:55.234940 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:34:55.236231 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:34:55.236231 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:34:55.236231 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:34:55.236231 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:34:55.236231 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:34:55.236231 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:34:55.236231 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:34:55.236231 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:34:55.236231 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:34:55.236231 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:34:55.236231 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:34:55.236231 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:34:55.236231 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:34:55.236231 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:34:55.236231 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:34:55.236231 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:34:55.236231 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:34:55.237233 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:34:55.237233 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:34:55.237233 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:34:55.237416 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 629/1237] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:34:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 300 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:34:55.243629 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae73033-feb5-4a3e-a43d-406748becd73 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:34:55.405093 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae73033-feb5-4a3e-a43d-406748becd73 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:34:55.422643 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:34:55.471171 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff01f2f-b963-48fc-b9b8-015157381733 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:34:55.472377 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff01f2f-b963-48fc-b9b8-015157381733 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:55.472838 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff01f2f-b963-48fc-b9b8-015157381733 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:55.473232 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff01f2f-b963-48fc-b9b8-015157381733 None admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:55.474220 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff01f2f-b963-48fc-b9b8-015157381733 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:34:55.474868 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff01f2f-b963-48fc-b9b8-015157381733 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:34:55.477002 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff01f2f-b963-48fc-b9b8-015157381733 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:34:55.498543 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 609/1238] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:34:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:34:55.506334 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d7a4791-8fd0-47c0-b653-31c0fc091331 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:34:55.662261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d7a4791-8fd0-47c0-b653-31c0fc091331 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:34:55.706042 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc9c8069-3ef0-42fa-b497-7e5eb2aae7ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:34:55.707031 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9c8069-3ef0-42fa-b497-7e5eb2aae7ba None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:55.707368 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9c8069-3ef0-42fa-b497-7e5eb2aae7ba None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:55.707820 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9c8069-3ef0-42fa-b497-7e5eb2aae7ba None admin] PATH_INFO: `/v3/users/placement` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:55.711795 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:34:55.718832 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc9c8069-3ef0-42fa-b497-7e5eb2aae7ba None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:34:55.720567 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc9c8069-3ef0-42fa-b497-7e5eb2aae7ba None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:34:55.736918 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-cc9c8069-3ef0-42fa-b497-7e5eb2aae7ba None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 28 09:34:55.736918 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:34:55.736918 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:34:55.736918 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:34:55.736918 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:34:55.736918 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:34:55.736918 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:34:55.736918 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:34:55.736918 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:34:55.736918 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:34:55.736918 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:34:55.736918 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:34:55.736918 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 09:34:55.736918 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 09:34:55.736918 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 09:34:55.736918 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 09:34:55.739306 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:34:55.739306 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:34:55.739306 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 09:34:55.739306 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:34:55.739306 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 09:34:55.739306 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:34:55.739306 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:34:55.739306 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:34:55.739306 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:34:55.739306 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:34:55.739306 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:34:55.739306 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:34:55.739306 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:34:55.739306 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:34:55.739306 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:34:55.739306 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:34:55.739306 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:34:55.739306 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:34:55.740448 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:34:55.740448 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:34:55.740448 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 09:34:55.740448 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 09:34:55.740448 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 09:34:55.740448 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 09:34:55.740448 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 09:34:55.740448 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 09:34:55.740448 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 28 09:34:55.740448 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:34:55.740448 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 630/1239] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:34:55 2022] GET /identity/v3/users/placement => generated 87 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:34:55.748433 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff01f2f-b963-48fc-b9b8-015157381733 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:34:55.881520 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff01f2f-b963-48fc-b9b8-015157381733 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:34:55.925627 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf759af-8ffa-4a8a-812e-9f961ad92021 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:34:55.926533 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf759af-8ffa-4a8a-812e-9f961ad92021 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:55.926935 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf759af-8ffa-4a8a-812e-9f961ad92021 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:55.927292 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf759af-8ffa-4a8a-812e-9f961ad92021 None admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:55.928232 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf759af-8ffa-4a8a-812e-9f961ad92021 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:34:55.928710 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf759af-8ffa-4a8a-812e-9f961ad92021 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:34:55.930658 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf759af-8ffa-4a8a-812e-9f961ad92021 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:34:55.959670 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 610/1240] 10.222.0.25 () {56 vars in 1143 bytes} [Fri Oct 28 09:34:55 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:34:55.961519 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc9c8069-3ef0-42fa-b497-7e5eb2aae7ba None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:34:56.114493 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc9c8069-3ef0-42fa-b497-7e5eb2aae7ba None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:34:56.171139 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816a76e5-e947-44c8-a5a8-7c8f369b7458 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:34:56.172025 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816a76e5-e947-44c8-a5a8-7c8f369b7458 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:56.172461 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816a76e5-e947-44c8-a5a8-7c8f369b7458 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:56.172878 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816a76e5-e947-44c8-a5a8-7c8f369b7458 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:56.266995 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:34:56.278763 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816a76e5-e947-44c8-a5a8-7c8f369b7458 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:34:56.280273 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816a76e5-e947-44c8-a5a8-7c8f369b7458 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:34:56.418776 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 631/1241] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:34:55 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 458 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 09:34:56.424818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf759af-8ffa-4a8a-812e-9f961ad92021 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:34:56.468573 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:34:56.596294 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf759af-8ffa-4a8a-812e-9f961ad92021 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:34:56.645498 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-481002bb-881a-45e3-a868-9480345f661c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:34:56.646581 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481002bb-881a-45e3-a868-9480345f661c None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:56.647069 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481002bb-881a-45e3-a868-9480345f661c None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:56.647525 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481002bb-881a-45e3-a868-9480345f661c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:56.663500 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-481002bb-881a-45e3-a868-9480345f661c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:34:56.665341 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-481002bb-881a-45e3-a868-9480345f661c None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:34:56.690049 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-481002bb-881a-45e3-a868-9480345f661c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:34:56.690049 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:34:56.690049 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:34:56.690049 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:34:56.690049 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:34:56.690049 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:34:56.690049 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:34:56.690049 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:34:56.690049 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:34:56.690049 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:34:56.690049 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:34:56.690049 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:34:56.690049 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:34:56.690049 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:34:56.690049 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:34:56.690049 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:34:56.692678 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:34:56.692678 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:34:56.692678 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:34:56.692678 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:34:56.692678 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:34:56.692678 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:34:56.692678 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:34:56.692678 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:34:56.692678 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:34:56.692678 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:34:56.692678 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:34:56.692678 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:34:56.692678 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:34:56.692678 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:34:56.692678 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:34:56.692678 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:34:56.692678 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:34:56.693812 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:34:56.693812 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:34:56.693812 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:34:56.693812 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 611/1242] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:34:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 269 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:34:56.701612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816a76e5-e947-44c8-a5a8-7c8f369b7458 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:34:56.845838 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:34:56.856491 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816a76e5-e947-44c8-a5a8-7c8f369b7458 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:34:56.909247 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-153bd9f6-85cf-441c-9c84-44409bb47e0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:34:56.910322 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153bd9f6-85cf-441c-9c84-44409bb47e0e None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:56.911070 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153bd9f6-85cf-441c-9c84-44409bb47e0e None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:56.911577 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153bd9f6-85cf-441c-9c84-44409bb47e0e None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:56.912629 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-153bd9f6-85cf-441c-9c84-44409bb47e0e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:34:56.913042 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-153bd9f6-85cf-441c-9c84-44409bb47e0e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:34:56.914603 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-153bd9f6-85cf-441c-9c84-44409bb47e0e None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:34:56.933595 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 632/1243] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:34:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:34:56.943056 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-481002bb-881a-45e3-a868-9480345f661c None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:34:57.062202 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:34:57.094309 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-481002bb-881a-45e3-a868-9480345f661c None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:34:57.160382 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9959e72-2f47-463c-a6d9-fa2a6572a115 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:34:57.162137 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9959e72-2f47-463c-a6d9-fa2a6572a115 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:57.162765 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9959e72-2f47-463c-a6d9-fa2a6572a115 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:57.163352 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9959e72-2f47-463c-a6d9-fa2a6572a115 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:57.175904 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9959e72-2f47-463c-a6d9-fa2a6572a115 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:34:57.177507 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9959e72-2f47-463c-a6d9-fa2a6572a115 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:34:57.195308 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-b9959e72-2f47-463c-a6d9-fa2a6572a115 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:34:57.195308 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:34:57.195308 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:34:57.195308 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:34:57.195308 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:34:57.195308 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:34:57.195308 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:34:57.195308 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:34:57.195308 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:34:57.195308 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:34:57.195308 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:34:57.195308 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:34:57.195308 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 09:34:57.195308 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 09:34:57.195308 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 09:34:57.196981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 09:34:57.196981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:34:57.196981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:34:57.196981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:34:57.196981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:34:57.196981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:34:57.196981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:34:57.196981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:34:57.196981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:34:57.196981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:34:57.196981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:34:57.196981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:34:57.196981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:34:57.196981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:34:57.196981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:34:57.196981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:34:57.196981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:34:57.199191 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 09:34:57.199191 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 09:34:57.199191 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 09:34:57.199191 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 09:34:57.199191 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 09:34:57.199191 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 09:34:57.199191 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:34:57.199191 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:34:57.199191 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 612/1244] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:34:56 2022] GET /identity/v3/projects/service => generated 88 bytes in 255 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:34:57.207502 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-153bd9f6-85cf-441c-9c84-44409bb47e0e None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:34:57.356842 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-153bd9f6-85cf-441c-9c84-44409bb47e0e None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:34:57.404454 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073d6d74-fbcf-474a-bd7e-eef73734b145 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:34:57.405355 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073d6d74-fbcf-474a-bd7e-eef73734b145 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:57.405710 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073d6d74-fbcf-474a-bd7e-eef73734b145 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:57.406062 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073d6d74-fbcf-474a-bd7e-eef73734b145 None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:57.406988 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-073d6d74-fbcf-474a-bd7e-eef73734b145 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:34:57.407453 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-073d6d74-fbcf-474a-bd7e-eef73734b145 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:34:57.409250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-073d6d74-fbcf-474a-bd7e-eef73734b145 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:34:57.425453 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 633/1245] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Oct 28 09:34:57 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:34:57.431568 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9959e72-2f47-463c-a6d9-fa2a6572a115 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:34:57.595285 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9959e72-2f47-463c-a6d9-fa2a6572a115 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:34:57.653736 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c21d8b67-bf36-4f4f-b35f-307e0e2d7aad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:34:57.654514 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21d8b67-bf36-4f4f-b35f-307e0e2d7aad None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:34:57.654983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21d8b67-bf36-4f4f-b35f-307e0e2d7aad None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:34:57.655479 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21d8b67-bf36-4f4f-b35f-307e0e2d7aad None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:34:57.656418 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c21d8b67-bf36-4f4f-b35f-307e0e2d7aad None admin] RBAC: Adding query filter params (role.id=e5d8d090857a4a018d0ec3c79ece6472, scope.project.id=455d1c826b974bc38497d18a516867c7, user.id=3f1d95049d2d4593bf52670b1a0035fa) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:34:57.656936 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c21d8b67-bf36-4f4f-b35f-307e0e2d7aad None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:34:57.658953 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c21d8b67-bf36-4f4f-b35f-307e0e2d7aad None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:34:57.670367 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 613/1246] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:34:57 2022] GET /identity/v3/role_assignments?user.id=3f1d95049d2d4593bf52670b1a0035fa&scope.project.id=455d1c826b974bc38497d18a516867c7&role.id=e5d8d090857a4a018d0ec3c79ece6472 => generated 608 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:35:00.002360 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6725562a-030e-464d-9c69-31a4657511eb None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:00.003115 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6725562a-030e-464d-9c69-31a4657511eb None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:00.003115 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6725562a-030e-464d-9c69-31a4657511eb None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:00.005074 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 634/1247] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:35:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 09:35:00.011319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cb1355-2c80-4939-ae8e-ce3ab3766eca None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:00.011874 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cb1355-2c80-4939-ae8e-ce3ab3766eca None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:00.011874 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cb1355-2c80-4939-ae8e-ce3ab3766eca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:00.018735 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:35:00.035356 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-b4cb1355-2c80-4939-ae8e-ce3ab3766eca None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:35:00.036948 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-b4cb1355-2c80-4939-ae8e-ce3ab3766eca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:35:00.137767 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 614/1248] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:35:00 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:35:00.449502 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6725562a-030e-464d-9c69-31a4657511eb None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:00.485776 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:35:00.594360 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6725562a-030e-464d-9c69-31a4657511eb None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:00.634045 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8428ce-6990-4481-9bcd-a9127128793b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:00.634715 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8428ce-6990-4481-9bcd-a9127128793b None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:00.634992 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8428ce-6990-4481-9bcd-a9127128793b None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:00.635269 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8428ce-6990-4481-9bcd-a9127128793b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:00.646419 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a8428ce-6990-4481-9bcd-a9127128793b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:35:00.651255 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-2a8428ce-6990-4481-9bcd-a9127128793b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:35:00.651255 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:35:00.651255 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:35:00.651255 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:35:00.651255 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:35:00.651255 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:35:00.651255 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:35:00.651255 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:35:00.651255 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:35:00.651255 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:35:00.651255 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:35:00.651255 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:35:00.651255 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:35:00.651255 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:35:00.651255 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:35:00.651255 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:35:00.652633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:35:00.652633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:35:00.652633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:35:00.652633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:35:00.652633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:35:00.652633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:35:00.652633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:35:00.652633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:35:00.652633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:35:00.652633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:35:00.652633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:35:00.652633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:35:00.652633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:35:00.652633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:35:00.652633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:35:00.652633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:35:00.654531 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:35:00.654531 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:35:00.654531 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:35:00.654531 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:35:00.654531 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:35:00.654531 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:35:00.654531 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:35:00.654531 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:35:00.654531 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:35:00.654531 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:35:00.654531 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:35:00.654531 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:35:00.654531 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 635/1249] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Oct 28 09:35:00 2022] GET /identity/v3/roles/admin => generated 83 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:35:00.658751 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4cb1355-2c80-4939-ae8e-ce3ab3766eca None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:00.796575 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4cb1355-2c80-4939-ae8e-ce3ab3766eca None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:00.849191 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:35:00.856323 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5161e7-7abc-4576-84fe-d04def0d713b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:00.857189 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5161e7-7abc-4576-84fe-d04def0d713b None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:00.857488 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5161e7-7abc-4576-84fe-d04def0d713b None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:00.857742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5161e7-7abc-4576-84fe-d04def0d713b None admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:00.858486 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5161e7-7abc-4576-84fe-d04def0d713b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:35:00.858984 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5161e7-7abc-4576-84fe-d04def0d713b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:35:00.860432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5161e7-7abc-4576-84fe-d04def0d713b None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:35:00.870831 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 615/1250] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:35:00 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:35:00.877575 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8428ce-6990-4481-9bcd-a9127128793b None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:00.967653 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:35:00.994536 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8428ce-6990-4481-9bcd-a9127128793b None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:01.046451 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab66330e-94cd-488a-927b-4b51a879a4b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:01.047250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab66330e-94cd-488a-927b-4b51a879a4b7 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:01.047375 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab66330e-94cd-488a-927b-4b51a879a4b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:01.047616 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab66330e-94cd-488a-927b-4b51a879a4b7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:01.061252 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab66330e-94cd-488a-927b-4b51a879a4b7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:35:01.063482 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab66330e-94cd-488a-927b-4b51a879a4b7 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:35:01.084773 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-ab66330e-94cd-488a-927b-4b51a879a4b7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:35:01.084773 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:35:01.084773 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:35:01.084773 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:35:01.084773 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:35:01.084773 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:35:01.084773 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:35:01.084773 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:35:01.084773 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:35:01.084773 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:35:01.084773 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:35:01.084773 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:35:01.084773 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:35:01.084773 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:35:01.084773 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:35:01.084773 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:35:01.087037 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:35:01.087037 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:35:01.087037 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:35:01.087037 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:35:01.087037 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:35:01.087037 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:35:01.087037 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:35:01.087037 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:35:01.087037 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:35:01.087037 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:35:01.087037 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:35:01.087037 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:35:01.087037 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:35:01.087037 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:35:01.087037 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:35:01.087037 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:35:01.087037 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:35:01.088927 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:35:01.088927 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:35:01.088927 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:35:01.088927 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 636/1251] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:35:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:35:01.092498 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5161e7-7abc-4576-84fe-d04def0d713b None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:01.250284 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5161e7-7abc-4576-84fe-d04def0d713b None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:01.309765 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7144451c-95bf-41d2-851b-cd5e44e189cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:01.310672 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7144451c-95bf-41d2-851b-cd5e44e189cb None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:01.311136 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7144451c-95bf-41d2-851b-cd5e44e189cb None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:01.311487 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7144451c-95bf-41d2-851b-cd5e44e189cb None admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:01.312511 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7144451c-95bf-41d2-851b-cd5e44e189cb None admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:35:01.312881 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7144451c-95bf-41d2-851b-cd5e44e189cb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:35:01.314804 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7144451c-95bf-41d2-851b-cd5e44e189cb None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:35:01.344032 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 616/1252] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:35:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:35:01.345524 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab66330e-94cd-488a-927b-4b51a879a4b7 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:01.514301 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab66330e-94cd-488a-927b-4b51a879a4b7 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:01.548303 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:35:01.559323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295d42ad-2116-4ca2-b9c5-43610c55ce27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:01.560044 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295d42ad-2116-4ca2-b9c5-43610c55ce27 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:01.560294 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295d42ad-2116-4ca2-b9c5-43610c55ce27 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:01.560503 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295d42ad-2116-4ca2-b9c5-43610c55ce27 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:01.569538 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-295d42ad-2116-4ca2-b9c5-43610c55ce27 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:35:01.573419 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-295d42ad-2116-4ca2-b9c5-43610c55ce27 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:35:01.586686 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-295d42ad-2116-4ca2-b9c5-43610c55ce27 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 28 09:35:01.586686 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:35:01.586686 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:35:01.586686 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:35:01.586686 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:35:01.586686 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:35:01.586686 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:35:01.586686 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:35:01.586686 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:35:01.586686 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:35:01.586686 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:35:01.586686 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:35:01.586686 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 09:35:01.586686 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 09:35:01.586686 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 09:35:01.586686 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 09:35:01.587894 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:35:01.587894 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:35:01.587894 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 09:35:01.587894 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:35:01.587894 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 09:35:01.587894 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:35:01.587894 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:35:01.587894 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:35:01.587894 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:35:01.587894 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:35:01.587894 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:35:01.587894 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:35:01.587894 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:35:01.587894 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:35:01.587894 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:35:01.587894 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:35:01.587894 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:35:01.587894 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:35:01.588962 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:35:01.588962 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:35:01.588962 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 09:35:01.588962 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 09:35:01.588962 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 09:35:01.588962 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 09:35:01.588962 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 09:35:01.588962 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 09:35:01.588962 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 28 09:35:01.588962 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:35:01.588962 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 637/1253] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:35:01 2022] GET /identity/v3/users/placement => generated 87 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:35:01.594181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7144451c-95bf-41d2-851b-cd5e44e189cb None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:01.667859 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:35:01.725884 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7144451c-95bf-41d2-851b-cd5e44e189cb None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:01.775427 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ddb168d-631d-4c8f-9b0c-109e9eb1924f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:01.776053 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddb168d-631d-4c8f-9b0c-109e9eb1924f None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:01.776332 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddb168d-631d-4c8f-9b0c-109e9eb1924f None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:01.776606 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddb168d-631d-4c8f-9b0c-109e9eb1924f None admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:01.777421 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ddb168d-631d-4c8f-9b0c-109e9eb1924f None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:35:01.777664 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ddb168d-631d-4c8f-9b0c-109e9eb1924f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:35:01.779181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ddb168d-631d-4c8f-9b0c-109e9eb1924f None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:35:01.800219 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 617/1254] 10.222.0.25 () {56 vars in 1143 bytes} [Fri Oct 28 09:35:01 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:35:01.806977 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295d42ad-2116-4ca2-b9c5-43610c55ce27 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:01.952784 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295d42ad-2116-4ca2-b9c5-43610c55ce27 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:01.993065 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8477d57-d4d4-4ec8-9261-6b64a7aa3f93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:01.993929 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8477d57-d4d4-4ec8-9261-6b64a7aa3f93 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:01.994317 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8477d57-d4d4-4ec8-9261-6b64a7aa3f93 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:01.994704 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8477d57-d4d4-4ec8-9261-6b64a7aa3f93 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:02.063165 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:35:02.081522 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8477d57-d4d4-4ec8-9261-6b64a7aa3f93 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:35:02.085160 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8477d57-d4d4-4ec8-9261-6b64a7aa3f93 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:35:02.256338 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 638/1255] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:35:01 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 450 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 09:35:02.262961 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ddb168d-631d-4c8f-9b0c-109e9eb1924f None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:02.411915 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ddb168d-631d-4c8f-9b0c-109e9eb1924f None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:02.439790 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:35:02.477081 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93165c43-bf04-4cf6-8805-75d7fcd06324 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:02.477930 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93165c43-bf04-4cf6-8805-75d7fcd06324 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:02.478309 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93165c43-bf04-4cf6-8805-75d7fcd06324 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:02.478745 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93165c43-bf04-4cf6-8805-75d7fcd06324 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:02.493645 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93165c43-bf04-4cf6-8805-75d7fcd06324 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:35:02.495928 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93165c43-bf04-4cf6-8805-75d7fcd06324 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:35:02.517014 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-93165c43-bf04-4cf6-8805-75d7fcd06324 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:35:02.517014 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:35:02.517014 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:35:02.517014 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:35:02.517014 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:35:02.517014 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:35:02.517014 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:35:02.517014 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:35:02.517014 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:35:02.517014 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:35:02.517014 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:35:02.517014 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:35:02.517014 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:35:02.517014 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:35:02.517014 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:35:02.517014 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:35:02.519024 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:35:02.519024 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:35:02.519024 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:35:02.519024 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:35:02.519024 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:35:02.519024 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:35:02.519024 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:35:02.519024 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:35:02.519024 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:35:02.519024 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:35:02.519024 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:35:02.519024 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:35:02.519024 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:35:02.519024 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:35:02.519024 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:35:02.519024 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:35:02.519024 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:35:02.521355 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:35:02.521355 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:35:02.521355 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:35:02.521355 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 618/1256] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:35:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 257 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:35:02.527826 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8477d57-d4d4-4ec8-9261-6b64a7aa3f93 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:02.669155 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:35:02.690303 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8477d57-d4d4-4ec8-9261-6b64a7aa3f93 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:02.751077 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0a7166-7ab7-4c79-b616-0748bc46afdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:02.752256 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0a7166-7ab7-4c79-b616-0748bc46afdd None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:02.752624 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0a7166-7ab7-4c79-b616-0748bc46afdd None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:02.753000 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0a7166-7ab7-4c79-b616-0748bc46afdd None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:02.754013 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0a7166-7ab7-4c79-b616-0748bc46afdd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:35:02.754576 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0a7166-7ab7-4c79-b616-0748bc46afdd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:35:02.757295 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0a7166-7ab7-4c79-b616-0748bc46afdd None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:35:02.775561 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 639/1257] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:35:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:35:02.782974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93165c43-bf04-4cf6-8805-75d7fcd06324 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:02.939497 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93165c43-bf04-4cf6-8805-75d7fcd06324 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:02.984938 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95752e05-7344-49da-8efa-a8fa408f831d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:02.985716 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95752e05-7344-49da-8efa-a8fa408f831d None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:02.986091 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95752e05-7344-49da-8efa-a8fa408f831d None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:02.986520 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95752e05-7344-49da-8efa-a8fa408f831d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:02.995500 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95752e05-7344-49da-8efa-a8fa408f831d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:35:02.997267 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95752e05-7344-49da-8efa-a8fa408f831d None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:35:03.010922 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-95752e05-7344-49da-8efa-a8fa408f831d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:35:03.010922 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:35:03.010922 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:35:03.010922 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:35:03.010922 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:35:03.010922 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:35:03.010922 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:35:03.010922 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:35:03.010922 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:35:03.010922 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:35:03.010922 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:35:03.010922 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:35:03.010922 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 09:35:03.010922 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 09:35:03.010922 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 09:35:03.012647 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 09:35:03.012647 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:35:03.012647 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:35:03.012647 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:35:03.012647 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:35:03.012647 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:35:03.012647 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:35:03.012647 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:35:03.012647 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:35:03.012647 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:35:03.012647 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:35:03.012647 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:35:03.012647 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:35:03.012647 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:35:03.012647 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:35:03.012647 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:35:03.012647 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:35:03.014137 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 09:35:03.014137 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 09:35:03.014137 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 09:35:03.014137 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 09:35:03.014137 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 09:35:03.014137 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 09:35:03.014137 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:35:03.014137 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:35:03.014137 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 619/1258] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:35:02 2022] GET /identity/v3/projects/service => generated 88 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:35:03.018545 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0a7166-7ab7-4c79-b616-0748bc46afdd None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:03.146353 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0a7166-7ab7-4c79-b616-0748bc46afdd None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:03.204110 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b2bd1c-2b10-489b-ab88-095479bb504b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:03.205105 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b2bd1c-2b10-489b-ab88-095479bb504b None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:03.205569 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b2bd1c-2b10-489b-ab88-095479bb504b None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:03.206015 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b2bd1c-2b10-489b-ab88-095479bb504b None admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:03.207345 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44b2bd1c-2b10-489b-ab88-095479bb504b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:35:03.207845 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44b2bd1c-2b10-489b-ab88-095479bb504b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:35:03.210386 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44b2bd1c-2b10-489b-ab88-095479bb504b None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:35:03.234534 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 640/1259] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Oct 28 09:35:03 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:35:03.242253 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95752e05-7344-49da-8efa-a8fa408f831d None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:03.279791 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:35:03.390257 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95752e05-7344-49da-8efa-a8fa408f831d None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:03.442791 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478e8d0b-1199-40f8-be3d-925086783953 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:03.443388 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478e8d0b-1199-40f8-be3d-925086783953 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:03.443728 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478e8d0b-1199-40f8-be3d-925086783953 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:03.444173 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478e8d0b-1199-40f8-be3d-925086783953 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:03.445445 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-478e8d0b-1199-40f8-be3d-925086783953 None admin] RBAC: Adding query filter params (role.id=e13d798a98ea4c3dbb87a236ec683d32, scope.project.id=455d1c826b974bc38497d18a516867c7, user.id=3f1d95049d2d4593bf52670b1a0035fa) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:35:03.445928 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-478e8d0b-1199-40f8-be3d-925086783953 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:35:03.448444 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-478e8d0b-1199-40f8-be3d-925086783953 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:35:03.456420 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 620/1260] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:35:03 2022] GET /identity/v3/role_assignments?user.id=3f1d95049d2d4593bf52670b1a0035fa&scope.project.id=455d1c826b974bc38497d18a516867c7&role.id=e13d798a98ea4c3dbb87a236ec683d32 => generated 259 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:35:05.689633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b710f01d-9e98-4742-bab2-cd51ebfb89b8 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:05.690210 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b710f01d-9e98-4742-bab2-cd51ebfb89b8 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:05.690210 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b710f01d-9e98-4742-bab2-cd51ebfb89b8 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:05.692148 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 641/1261] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:35:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:35:05.698116 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6950dcc5-bd8e-4bce-8902-7e6cf5636414 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:05.698662 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6950dcc5-bd8e-4bce-8902-7e6cf5636414 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:05.698754 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6950dcc5-bd8e-4bce-8902-7e6cf5636414 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:05.717935 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-6950dcc5-bd8e-4bce-8902-7e6cf5636414 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:35:05.719164 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-6950dcc5-bd8e-4bce-8902-7e6cf5636414 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:35:05.791821 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 621/1262] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:35:05 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 09:35:06.056417 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b710f01d-9e98-4742-bab2-cd51ebfb89b8 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:06.067789 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:35:06.186538 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b710f01d-9e98-4742-bab2-cd51ebfb89b8 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:06.234837 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d16f3fc1-2959-439d-85cf-92f6d06e9bfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:06.235537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16f3fc1-2959-439d-85cf-92f6d06e9bfa None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:06.235863 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16f3fc1-2959-439d-85cf-92f6d06e9bfa None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:06.236184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16f3fc1-2959-439d-85cf-92f6d06e9bfa None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:06.247582 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d16f3fc1-2959-439d-85cf-92f6d06e9bfa None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:35:06.252503 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-d16f3fc1-2959-439d-85cf-92f6d06e9bfa None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:35:06.252503 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:35:06.252503 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:35:06.252503 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:35:06.252503 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:35:06.252503 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:35:06.252503 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:35:06.252503 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:35:06.252503 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:35:06.252503 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:35:06.252503 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:35:06.252503 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:35:06.252503 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:35:06.252503 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:35:06.252503 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:35:06.252503 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:35:06.254612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:35:06.254612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:35:06.254612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:35:06.254612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:35:06.254612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:35:06.254612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:35:06.254612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:35:06.254612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:35:06.254612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:35:06.254612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:35:06.254612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:35:06.254612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:35:06.254612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:35:06.254612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:35:06.254612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:35:06.254612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:35:06.255659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:35:06.255659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:35:06.255659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:35:06.255659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:35:06.255659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:35:06.255659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:35:06.255659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:35:06.255659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:35:06.255659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:35:06.255659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:35:06.255659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:35:06.255659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:35:06.255659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 642/1263] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Oct 28 09:35:06 2022] GET /identity/v3/roles/admin => generated 83 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:35:06.261251 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6950dcc5-bd8e-4bce-8902-7e6cf5636414 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:06.280871 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:35:06.370556 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6950dcc5-bd8e-4bce-8902-7e6cf5636414 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:06.408311 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50c56c9-2d74-4cac-8d27-6ea2a825081c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:06.408940 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50c56c9-2d74-4cac-8d27-6ea2a825081c None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:06.409221 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50c56c9-2d74-4cac-8d27-6ea2a825081c None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:06.409495 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50c56c9-2d74-4cac-8d27-6ea2a825081c None admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:06.411026 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f50c56c9-2d74-4cac-8d27-6ea2a825081c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:35:06.411335 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f50c56c9-2d74-4cac-8d27-6ea2a825081c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:35:06.412780 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f50c56c9-2d74-4cac-8d27-6ea2a825081c None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:35:06.423701 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 622/1264] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:35:06 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:35:06.430331 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d16f3fc1-2959-439d-85cf-92f6d06e9bfa None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:06.502698 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:35:06.538777 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d16f3fc1-2959-439d-85cf-92f6d06e9bfa None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:06.574591 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472a55e4-cb10-44e1-913b-7227c476005f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:06.575555 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472a55e4-cb10-44e1-913b-7227c476005f None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:06.575555 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472a55e4-cb10-44e1-913b-7227c476005f None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:06.575739 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472a55e4-cb10-44e1-913b-7227c476005f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:06.587286 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-472a55e4-cb10-44e1-913b-7227c476005f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:35:06.588742 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-472a55e4-cb10-44e1-913b-7227c476005f None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:35:06.604187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-472a55e4-cb10-44e1-913b-7227c476005f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:35:06.604187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:35:06.604187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:35:06.604187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:35:06.604187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:35:06.604187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:35:06.604187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:35:06.604187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:35:06.604187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:35:06.604187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:35:06.604187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:35:06.604187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:35:06.604187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:35:06.604187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:35:06.604187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:35:06.604187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:35:06.605537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:35:06.605537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:35:06.605537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:35:06.605537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:35:06.605537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:35:06.605537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:35:06.605537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:35:06.605537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:35:06.605537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:35:06.605537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:35:06.605537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:35:06.605537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:35:06.605537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:35:06.605537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:35:06.605537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:35:06.605537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:35:06.605537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:35:06.606720 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:35:06.606720 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:35:06.606720 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:35:06.606720 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 643/1265] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:35:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:35:06.611363 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50c56c9-2d74-4cac-8d27-6ea2a825081c None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:06.701239 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:35:06.723483 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50c56c9-2d74-4cac-8d27-6ea2a825081c None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:06.765272 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b66c70-49ee-4898-a7fb-18e90798ec3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:06.765891 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b66c70-49ee-4898-a7fb-18e90798ec3d None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:06.766174 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b66c70-49ee-4898-a7fb-18e90798ec3d None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:06.766478 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b66c70-49ee-4898-a7fb-18e90798ec3d None admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:06.767129 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b66c70-49ee-4898-a7fb-18e90798ec3d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:35:06.767436 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b66c70-49ee-4898-a7fb-18e90798ec3d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:35:06.768815 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b66c70-49ee-4898-a7fb-18e90798ec3d None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:35:06.782124 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 623/1266] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:35:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:35:06.788650 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472a55e4-cb10-44e1-913b-7227c476005f None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:06.913753 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472a55e4-cb10-44e1-913b-7227c476005f None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:06.937482 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:35:06.952202 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e448b404-e1b1-42d0-b57b-db0bd4fed268 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:06.952758 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e448b404-e1b1-42d0-b57b-db0bd4fed268 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:06.953040 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e448b404-e1b1-42d0-b57b-db0bd4fed268 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:06.953317 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e448b404-e1b1-42d0-b57b-db0bd4fed268 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:06.961302 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e448b404-e1b1-42d0-b57b-db0bd4fed268 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:35:06.962783 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e448b404-e1b1-42d0-b57b-db0bd4fed268 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:35:06.975116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-e448b404-e1b1-42d0-b57b-db0bd4fed268 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 28 09:35:06.975116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:35:06.975116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:35:06.975116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:35:06.975116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:35:06.975116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:35:06.975116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:35:06.975116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:35:06.975116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:35:06.975116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:35:06.975116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:35:06.975116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:35:06.975116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 09:35:06.975116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 09:35:06.975116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 09:35:06.975116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 09:35:06.976812 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:35:06.976812 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:35:06.976812 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 09:35:06.976812 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:35:06.976812 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 09:35:06.976812 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:35:06.976812 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:35:06.976812 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:35:06.976812 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:35:06.976812 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:35:06.976812 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:35:06.976812 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:35:06.976812 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:35:06.976812 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:35:06.976812 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:35:06.976812 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:35:06.976812 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:35:06.976812 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:35:06.977955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:35:06.977955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:35:06.977955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 09:35:06.977955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 09:35:06.977955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 09:35:06.977955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 09:35:06.977955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 09:35:06.977955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 09:35:06.977955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 28 09:35:06.977955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:35:06.977955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 644/1267] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:35:06 2022] GET /identity/v3/users/placement => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:35:06.982379 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b66c70-49ee-4898-a7fb-18e90798ec3d None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:07.085188 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b66c70-49ee-4898-a7fb-18e90798ec3d None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:07.122023 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e10f31-b726-40f0-9ac8-c02bf4104ef2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:07.122668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e10f31-b726-40f0-9ac8-c02bf4104ef2 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:07.122950 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e10f31-b726-40f0-9ac8-c02bf4104ef2 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:07.123263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e10f31-b726-40f0-9ac8-c02bf4104ef2 None admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:07.123964 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e10f31-b726-40f0-9ac8-c02bf4104ef2 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:35:07.124270 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e10f31-b726-40f0-9ac8-c02bf4104ef2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:35:07.125705 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e10f31-b726-40f0-9ac8-c02bf4104ef2 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:35:07.136311 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:35:07.147994 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 624/1268] 10.222.0.25 () {56 vars in 1143 bytes} [Fri Oct 28 09:35:06 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:35:07.154899 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e448b404-e1b1-42d0-b57b-db0bd4fed268 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:07.261947 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e448b404-e1b1-42d0-b57b-db0bd4fed268 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:07.296833 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d119af3-b6ae-42e8-8009-3e0d2d2e5e78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:07.297455 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d119af3-b6ae-42e8-8009-3e0d2d2e5e78 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:07.297733 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d119af3-b6ae-42e8-8009-3e0d2d2e5e78 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:07.298003 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d119af3-b6ae-42e8-8009-3e0d2d2e5e78 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:07.310382 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d119af3-b6ae-42e8-8009-3e0d2d2e5e78 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:35:07.312283 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d119af3-b6ae-42e8-8009-3e0d2d2e5e78 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:35:07.327285 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-1d119af3-b6ae-42e8-8009-3e0d2d2e5e78 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:35:07.327285 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:35:07.327285 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:35:07.327285 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:35:07.327285 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:35:07.327285 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:35:07.327285 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:35:07.327285 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:35:07.327285 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:35:07.327285 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:35:07.327285 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:35:07.327285 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:35:07.327285 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:35:07.327285 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:35:07.327285 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:35:07.327285 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:35:07.328955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:35:07.328955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:35:07.328955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:35:07.328955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:35:07.328955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:35:07.328955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:35:07.328955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:35:07.328955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:35:07.328955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:35:07.328955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:35:07.328955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:35:07.328955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:35:07.328955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:35:07.328955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:35:07.328955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:35:07.328955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:35:07.328955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:35:07.330621 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:35:07.330621 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:35:07.330621 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:35:07.330621 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 645/1269] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:35:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:35:07.334504 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e10f31-b726-40f0-9ac8-c02bf4104ef2 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:07.453234 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e10f31-b726-40f0-9ac8-c02bf4104ef2 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:07.492405 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a2d19e-72e2-4695-8a2d-463a212f79c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:07.493044 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a2d19e-72e2-4695-8a2d-463a212f79c2 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:07.493326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a2d19e-72e2-4695-8a2d-463a212f79c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:07.493602 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a2d19e-72e2-4695-8a2d-463a212f79c2 None admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:07.494273 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a2d19e-72e2-4695-8a2d-463a212f79c2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:35:07.494628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a2d19e-72e2-4695-8a2d-463a212f79c2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:35:07.496024 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a2d19e-72e2-4695-8a2d-463a212f79c2 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:35:07.509838 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 625/1270] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:35:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:35:07.516255 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d119af3-b6ae-42e8-8009-3e0d2d2e5e78 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:07.551501 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:35:07.625769 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d119af3-b6ae-42e8-8009-3e0d2d2e5e78 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:07.661110 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a46ec0e3-1147-4b60-acf2-bb36dab76319 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:07.661744 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a46ec0e3-1147-4b60-acf2-bb36dab76319 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:07.662019 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a46ec0e3-1147-4b60-acf2-bb36dab76319 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:07.662299 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a46ec0e3-1147-4b60-acf2-bb36dab76319 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:07.669565 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a46ec0e3-1147-4b60-acf2-bb36dab76319 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:35:07.671065 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a46ec0e3-1147-4b60-acf2-bb36dab76319 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:35:07.683135 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-a46ec0e3-1147-4b60-acf2-bb36dab76319 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:35:07.683135 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:35:07.683135 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:35:07.683135 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:35:07.683135 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:35:07.683135 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:35:07.683135 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:35:07.683135 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:35:07.683135 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:35:07.683135 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:35:07.683135 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:35:07.683135 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:35:07.683135 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 09:35:07.683135 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 09:35:07.683135 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 09:35:07.684298 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 09:35:07.684298 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:35:07.684298 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:35:07.684298 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:35:07.684298 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:35:07.684298 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:35:07.684298 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:35:07.684298 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:35:07.684298 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:35:07.684298 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:35:07.684298 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:35:07.684298 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:35:07.684298 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:35:07.684298 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:35:07.684298 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:35:07.684298 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:35:07.684298 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:35:07.686016 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 09:35:07.686016 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 09:35:07.686016 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 09:35:07.686016 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 09:35:07.686016 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 09:35:07.686016 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 09:35:07.686016 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:35:07.686016 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:35:07.686016 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 646/1271] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:35:07 2022] GET /identity/v3/projects/service => generated 88 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:35:07.691697 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a2d19e-72e2-4695-8a2d-463a212f79c2 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:07.758450 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:35:07.803704 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a2d19e-72e2-4695-8a2d-463a212f79c2 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:07.841502 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a4b7370-3442-44bc-8457-07ddb92e2cfb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:07.842112 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4b7370-3442-44bc-8457-07ddb92e2cfb None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:07.842428 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4b7370-3442-44bc-8457-07ddb92e2cfb None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:07.842695 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a4b7370-3442-44bc-8457-07ddb92e2cfb None admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:07.843400 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a4b7370-3442-44bc-8457-07ddb92e2cfb None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:35:07.843705 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a4b7370-3442-44bc-8457-07ddb92e2cfb None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:35:07.845152 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a4b7370-3442-44bc-8457-07ddb92e2cfb None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:35:07.857919 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 626/1272] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Oct 28 09:35:07 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:35:07.864360 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a46ec0e3-1147-4b60-acf2-bb36dab76319 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:07.974303 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:35:07.983416 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a46ec0e3-1147-4b60-acf2-bb36dab76319 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:08.018910 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0fb270c-1f03-4d48-b011-a063a6384f97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:08.019628 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fb270c-1f03-4d48-b011-a063a6384f97 None admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:08.019908 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fb270c-1f03-4d48-b011-a063a6384f97 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:08.020186 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fb270c-1f03-4d48-b011-a063a6384f97 None admin] PATH_INFO: `/v3/projects/455d1c826b974bc38497d18a516867c7/users/3f1d95049d2d4593bf52670b1a0035fa/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:08.022134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0fb270c-1f03-4d48-b011-a063a6384f97 None admin] RBAC: Authorizing `identity:create_grant(project_id=455d1c826b974bc38497d18a516867c7, user_id=3f1d95049d2d4593bf52670b1a0035fa, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:35:08.023622 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0fb270c-1f03-4d48-b011-a063a6384f97 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:35:08.038951 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 647/1273] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Oct 28 09:35:07 2022] PUT /identity/v3/projects/455d1c826b974bc38497d18a516867c7/users/3f1d95049d2d4593bf52670b1a0035fa/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:35:10.235871 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c217f6f-5812-4a24-a9aa-18aa406fad17 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:10.236845 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c217f6f-5812-4a24-a9aa-18aa406fad17 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:10.236845 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c217f6f-5812-4a24-a9aa-18aa406fad17 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:10.238927 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 627/1274] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:35:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:35:10.245090 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac2c58b-d50f-4148-9810-e72346c4391a None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:10.245671 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac2c58b-d50f-4148-9810-e72346c4391a None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:10.245671 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac2c58b-d50f-4148-9810-e72346c4391a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:10.265483 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-5ac2c58b-d50f-4148-9810-e72346c4391a None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:35:10.267109 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-5ac2c58b-d50f-4148-9810-e72346c4391a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:35:10.344904 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 648/1275] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:35:10 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:35:10.615065 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c217f6f-5812-4a24-a9aa-18aa406fad17 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:10.785175 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c217f6f-5812-4a24-a9aa-18aa406fad17 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:10.815807 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:35:10.836768 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a98ef625-21e9-49ce-9500-48008bbff9f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:10.837410 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98ef625-21e9-49ce-9500-48008bbff9f4 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:10.837693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98ef625-21e9-49ce-9500-48008bbff9f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:10.837969 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98ef625-21e9-49ce-9500-48008bbff9f4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:10.849491 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a98ef625-21e9-49ce-9500-48008bbff9f4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Oct 28 09:35:10.854374 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-a98ef625-21e9-49ce-9500-48008bbff9f4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:35:10.854374 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:35:10.854374 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:35:10.854374 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:35:10.854374 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:35:10.854374 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:35:10.854374 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:35:10.854374 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:35:10.854374 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:35:10.854374 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:35:10.854374 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:35:10.854374 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:35:10.854374 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Oct 28 09:35:10.854374 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(role_id) Oct 28 09:35:10.854374 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Oct 28 09:35:10.854374 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Oct 28 09:35:10.856122 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Oct 28 09:35:10.856122 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Oct 28 09:35:10.856122 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Oct 28 09:35:10.856122 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Oct 28 09:35:10.856122 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:35:10.856122 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:35:10.856122 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:35:10.856122 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:35:10.856122 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:35:10.856122 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:35:10.856122 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:35:10.856122 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:35:10.856122 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:35:10.856122 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:35:10.856122 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:35:10.856122 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:35:10.857717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:35:10.857717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:35:10.857717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:35:10.857717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:35:10.857717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Oct 28 09:35:10.857717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Oct 28 09:35:10.857717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Oct 28 09:35:10.857717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Oct 28 09:35:10.857717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Oct 28 09:35:10.857717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Oct 28 09:35:10.857717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Oct 28 09:35:10.857717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:35:10.857717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 628/1276] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Oct 28 09:35:10 2022] GET /identity/v3/roles/admin => generated 83 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:35:10.862031 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac2c58b-d50f-4148-9810-e72346c4391a None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:10.950501 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:35:10.972748 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac2c58b-d50f-4148-9810-e72346c4391a None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:11.012815 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a00e1da-8d11-4993-b9ca-5dd2ce598094 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:11.013428 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a00e1da-8d11-4993-b9ca-5dd2ce598094 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:11.013706 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a00e1da-8d11-4993-b9ca-5dd2ce598094 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:11.013978 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a00e1da-8d11-4993-b9ca-5dd2ce598094 None admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:11.014676 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a00e1da-8d11-4993-b9ca-5dd2ce598094 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:35:11.014981 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a00e1da-8d11-4993-b9ca-5dd2ce598094 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:35:11.016331 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a00e1da-8d11-4993-b9ca-5dd2ce598094 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:35:11.026837 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 649/1277] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Oct 28 09:35:10 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:35:11.032975 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a98ef625-21e9-49ce-9500-48008bbff9f4 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:11.179591 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a98ef625-21e9-49ce-9500-48008bbff9f4 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:11.231169 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329a1971-bcdc-43a8-842c-5d25f770f841 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:11.231800 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329a1971-bcdc-43a8-842c-5d25f770f841 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:11.232086 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329a1971-bcdc-43a8-842c-5d25f770f841 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:11.232352 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329a1971-bcdc-43a8-842c-5d25f770f841 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:11.244929 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-329a1971-bcdc-43a8-842c-5d25f770f841 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:35:11.246615 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-329a1971-bcdc-43a8-842c-5d25f770f841 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:35:11.264657 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-329a1971-bcdc-43a8-842c-5d25f770f841 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:35:11.264657 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:35:11.264657 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:35:11.264657 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:35:11.264657 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:35:11.264657 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:35:11.264657 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:35:11.264657 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:35:11.264657 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:35:11.264657 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:35:11.264657 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:35:11.264657 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:35:11.264657 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:35:11.264657 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:35:11.264657 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:35:11.264657 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:35:11.266499 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:35:11.266499 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:35:11.266499 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:35:11.266499 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:35:11.266499 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:35:11.266499 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:35:11.266499 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:35:11.266499 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:35:11.266499 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:35:11.266499 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:35:11.266499 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:35:11.266499 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:35:11.266499 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:35:11.266499 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:35:11.266499 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:35:11.266499 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:35:11.266499 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:35:11.268639 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:35:11.268639 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:35:11.268639 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:35:11.268639 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 629/1278] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:35:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:35:11.273648 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a00e1da-8d11-4993-b9ca-5dd2ce598094 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:11.435782 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a00e1da-8d11-4993-b9ca-5dd2ce598094 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:11.480401 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a247a02-7435-4e66-b894-683388f6960c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:11.481011 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a247a02-7435-4e66-b894-683388f6960c None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:11.481290 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a247a02-7435-4e66-b894-683388f6960c None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:11.481562 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a247a02-7435-4e66-b894-683388f6960c None admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:11.482204 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a247a02-7435-4e66-b894-683388f6960c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:35:11.482541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a247a02-7435-4e66-b894-683388f6960c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:35:11.483909 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a247a02-7435-4e66-b894-683388f6960c None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:35:11.498021 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 650/1279] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:35:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:35:11.504030 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329a1971-bcdc-43a8-842c-5d25f770f841 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:11.538125 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:35:11.616140 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329a1971-bcdc-43a8-842c-5d25f770f841 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:11.651925 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5653158-a0b1-4f5b-b02f-b3997b17f7e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:11.652560 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5653158-a0b1-4f5b-b02f-b3997b17f7e9 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:11.652843 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5653158-a0b1-4f5b-b02f-b3997b17f7e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:11.653118 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5653158-a0b1-4f5b-b02f-b3997b17f7e9 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:11.662452 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5653158-a0b1-4f5b-b02f-b3997b17f7e9 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:35:11.663910 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5653158-a0b1-4f5b-b02f-b3997b17f7e9 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:35:11.676167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-a5653158-a0b1-4f5b-b02f-b3997b17f7e9 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Oct 28 09:35:11.676167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:35:11.676167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:35:11.676167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:35:11.676167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:35:11.676167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:35:11.676167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:35:11.676167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:35:11.676167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:35:11.676167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:35:11.676167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:35:11.676167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:35:11.676167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Oct 28 09:35:11.676167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_user(user_id) Oct 28 09:35:11.676167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Oct 28 09:35:11.676167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Oct 28 09:35:11.677861 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:35:11.677861 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:35:11.677861 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Oct 28 09:35:11.677861 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:35:11.677861 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Oct 28 09:35:11.677861 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Oct 28 09:35:11.677861 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:35:11.677861 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:35:11.677861 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:35:11.677861 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:35:11.677861 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:35:11.677861 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:35:11.677861 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:35:11.677861 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:35:11.677861 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:35:11.677861 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:35:11.677861 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:35:11.677861 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:35:11.679665 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:35:11.679665 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:35:11.679665 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Oct 28 09:35:11.679665 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Oct 28 09:35:11.679665 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Oct 28 09:35:11.679665 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Oct 28 09:35:11.679665 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Oct 28 09:35:11.679665 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Oct 28 09:35:11.679665 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Oct 28 09:35:11.679665 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:35:11.679665 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 630/1280] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:35:11 2022] GET /identity/v3/users/placement => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:35:11.683633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a247a02-7435-4e66-b894-683388f6960c None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:11.688086 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:35:11.797219 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a247a02-7435-4e66-b894-683388f6960c None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:11.835356 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8700059b-542f-4c8a-996a-cc95232ca88c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:11.835974 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8700059b-542f-4c8a-996a-cc95232ca88c None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:11.836253 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8700059b-542f-4c8a-996a-cc95232ca88c None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:11.836523 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8700059b-542f-4c8a-996a-cc95232ca88c None admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:11.837253 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8700059b-542f-4c8a-996a-cc95232ca88c None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:35:11.837554 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8700059b-542f-4c8a-996a-cc95232ca88c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:35:11.838991 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8700059b-542f-4c8a-996a-cc95232ca88c None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:35:11.857784 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 651/1281] 10.222.0.25 () {56 vars in 1143 bytes} [Fri Oct 28 09:35:11 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:35:11.864286 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5653158-a0b1-4f5b-b02f-b3997b17f7e9 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:11.960665 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:35:11.971334 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5653158-a0b1-4f5b-b02f-b3997b17f7e9 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:12.008218 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f78cfa-c95e-49e8-9e57-c5e43f5204ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:12.008842 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f78cfa-c95e-49e8-9e57-c5e43f5204ae None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:12.009123 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f78cfa-c95e-49e8-9e57-c5e43f5204ae None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:12.009401 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f78cfa-c95e-49e8-9e57-c5e43f5204ae None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:12.083279 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73f78cfa-c95e-49e8-9e57-c5e43f5204ae None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:35:12.084741 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73f78cfa-c95e-49e8-9e57-c5e43f5204ae None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:35:12.181224 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:35:12.197317 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 631/1282] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Oct 28 09:35:11 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Oct 28 09:35:12.203381 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8700059b-542f-4c8a-996a-cc95232ca88c None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:12.270171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:35:12.311032 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8700059b-542f-4c8a-996a-cc95232ca88c None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:12.351591 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44108f2a-5ad4-4738-b888-28804bac0464 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:12.352662 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44108f2a-5ad4-4738-b888-28804bac0464 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:12.353145 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44108f2a-5ad4-4738-b888-28804bac0464 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:12.353630 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44108f2a-5ad4-4738-b888-28804bac0464 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:12.372200 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44108f2a-5ad4-4738-b888-28804bac0464 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:35:12.375019 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44108f2a-5ad4-4738-b888-28804bac0464 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:35:12.402155 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-44108f2a-5ad4-4738-b888-28804bac0464 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:35:12.402155 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:35:12.402155 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:35:12.402155 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:35:12.402155 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:35:12.402155 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:35:12.402155 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:35:12.402155 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:35:12.402155 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:35:12.402155 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:35:12.402155 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:35:12.402155 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:35:12.402155 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Oct 28 09:35:12.402155 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Oct 28 09:35:12.402155 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Oct 28 09:35:12.402155 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Oct 28 09:35:12.403408 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:35:12.403408 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:35:12.403408 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:35:12.403408 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:35:12.403408 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:35:12.403408 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:35:12.403408 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:35:12.403408 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:35:12.403408 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:35:12.403408 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:35:12.403408 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:35:12.403408 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:35:12.403408 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:35:12.403408 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:35:12.403408 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:35:12.403408 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:35:12.403408 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Oct 28 09:35:12.404525 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Oct 28 09:35:12.404525 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Oct 28 09:35:12.404525 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:35:12.404525 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 652/1283] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:35:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:35:12.413814 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f78cfa-c95e-49e8-9e57-c5e43f5204ae None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:12.517212 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f78cfa-c95e-49e8-9e57-c5e43f5204ae None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:12.557071 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a07b50c-268e-4f93-b1a4-0b93f4846a60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:12.557597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a07b50c-268e-4f93-b1a4-0b93f4846a60 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:12.557880 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a07b50c-268e-4f93-b1a4-0b93f4846a60 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:12.558156 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a07b50c-268e-4f93-b1a4-0b93f4846a60 None admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:12.558851 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a07b50c-268e-4f93-b1a4-0b93f4846a60 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:35:12.559158 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a07b50c-268e-4f93-b1a4-0b93f4846a60 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:35:12.560550 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a07b50c-268e-4f93-b1a4-0b93f4846a60 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:35:12.574181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 632/1284] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Oct 28 09:35:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:35:12.580224 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44108f2a-5ad4-4738-b888-28804bac0464 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:12.687422 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44108f2a-5ad4-4738-b888-28804bac0464 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:12.704527 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:35:12.728894 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7aaf3e2-5d40-4966-9e5d-ca10f6cc5898 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:12.729537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7aaf3e2-5d40-4966-9e5d-ca10f6cc5898 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:12.729815 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7aaf3e2-5d40-4966-9e5d-ca10f6cc5898 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:12.730085 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7aaf3e2-5d40-4966-9e5d-ca10f6cc5898 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:12.737593 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7aaf3e2-5d40-4966-9e5d-ca10f6cc5898 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:35:12.739602 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7aaf3e2-5d40-4966-9e5d-ca10f6cc5898 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:35:12.751654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application [None req-d7aaf3e2-5d40-4966-9e5d-ca10f6cc5898 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:35:12.751654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:35:12.751654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:35:12.751654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:35:12.751654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:35:12.751654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:35:12.751654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:35:12.751654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:35:12.751654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:35:12.751654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:35:12.751654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:35:12.751654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:35:12.751654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Oct 28 09:35:12.751654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(project_id) Oct 28 09:35:12.751654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Oct 28 09:35:12.753072 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Oct 28 09:35:12.753072 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:35:12.753072 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:35:12.753072 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:35:12.753072 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:35:12.753072 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:35:12.753072 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:35:12.753072 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:35:12.753072 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application with Lock( Oct 28 09:35:12.753072 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:35:12.753072 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:35:12.753072 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:35:12.753072 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:35:12.753072 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:35:12.753072 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:35:12.753072 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:35:12.753072 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:35:12.754123 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Oct 28 09:35:12.754123 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Oct 28 09:35:12.754123 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Oct 28 09:35:12.754123 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Oct 28 09:35:12.754123 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Oct 28 09:35:12.754123 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Oct 28 09:35:12.754123 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Oct 28 09:35:12.754123 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: ERROR keystone.server.flask.application Oct 28 09:35:12.754123 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 653/1285] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Oct 28 09:35:12 2022] GET /identity/v3/projects/service => generated 88 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:35:12.759049 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a07b50c-268e-4f93-b1a4-0b93f4846a60 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:12.814081 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:35:12.870369 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a07b50c-268e-4f93-b1a4-0b93f4846a60 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:12.905754 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f75b46-cb2c-424d-a9a6-73b2f7c30f87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:12.906375 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f75b46-cb2c-424d-a9a6-73b2f7c30f87 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:12.906691 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f75b46-cb2c-424d-a9a6-73b2f7c30f87 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:12.906969 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f75b46-cb2c-424d-a9a6-73b2f7c30f87 None admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:12.907678 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18f75b46-cb2c-424d-a9a6-73b2f7c30f87 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:35:12.907991 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18f75b46-cb2c-424d-a9a6-73b2f7c30f87 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:35:12.909434 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18f75b46-cb2c-424d-a9a6-73b2f7c30f87 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:35:12.923191 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 633/1286] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Oct 28 09:35:12 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:35:12.929615 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7aaf3e2-5d40-4966-9e5d-ca10f6cc5898 None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:13.038025 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7aaf3e2-5d40-4966-9e5d-ca10f6cc5898 None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:13.074106 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efe19bd1-8f97-4798-a598-06b80fbc554a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:13.074660 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe19bd1-8f97-4798-a598-06b80fbc554a None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:13.074944 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe19bd1-8f97-4798-a598-06b80fbc554a None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:13.075220 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efe19bd1-8f97-4798-a598-06b80fbc554a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:13.075956 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe19bd1-8f97-4798-a598-06b80fbc554a None admin] RBAC: Adding query filter params (role.id=e13d798a98ea4c3dbb87a236ec683d32, scope.project.id=455d1c826b974bc38497d18a516867c7, user.id=3f1d95049d2d4593bf52670b1a0035fa) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:35:13.076266 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe19bd1-8f97-4798-a598-06b80fbc554a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:35:13.077704 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efe19bd1-8f97-4798-a598-06b80fbc554a None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:35:13.084482 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 654/1287] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Oct 28 09:35:12 2022] GET /identity/v3/role_assignments?user.id=3f1d95049d2d4593bf52670b1a0035fa&scope.project.id=455d1c826b974bc38497d18a516867c7&role.id=e13d798a98ea4c3dbb87a236ec683d32 => generated 608 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:35:15.315830 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-154e5171-4ba8-4e1a-baf2-0cc2750b00a7 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:15.317081 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-154e5171-4ba8-4e1a-baf2-0cc2750b00a7 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:15.317081 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-154e5171-4ba8-4e1a-baf2-0cc2750b00a7 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:15.318251 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 634/1288] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:35:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:35:15.324538 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a9fa72-7432-46b7-8815-fc2e5bafb29f None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:15.325086 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a9fa72-7432-46b7-8815-fc2e5bafb29f None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:15.325190 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a9fa72-7432-46b7-8815-fc2e5bafb29f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:15.345437 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-60a9fa72-7432-46b7-8815-fc2e5bafb29f None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:35:15.346773 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-60a9fa72-7432-46b7-8815-fc2e5bafb29f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:35:15.379649 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:35:15.422727 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 655/1289] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:35:15 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:35:15.705420 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-154e5171-4ba8-4e1a-baf2-0cc2750b00a7 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:15.813935 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-154e5171-4ba8-4e1a-baf2-0cc2750b00a7 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:15.825163 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:35:15.850094 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c32525c5-26af-4519-9700-b56817888859 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:15.850744 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32525c5-26af-4519-9700-b56817888859 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:15.851022 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32525c5-26af-4519-9700-b56817888859 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:15.851291 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32525c5-26af-4519-9700-b56817888859 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:15.851874 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c32525c5-26af-4519-9700-b56817888859 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:35:15.853235 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c32525c5-26af-4519-9700-b56817888859 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:35:15.862432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-c32525c5-26af-4519-9700-b56817888859 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 28 09:35:15.862432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:35:15.862432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 09:35:15.862432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 09:35:15.862432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 09:35:15.862432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 09:35:15.862432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 09:35:15.862432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:35:15.862432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 28 09:35:15.862432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:35:15.862432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 09:35:15.862432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:35:15.862432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:35:15.862432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:35:15.862432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:35:15.862432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:35:15.862432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:35:15.864141 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:35:15.864141 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:35:15.864141 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:35:15.864141 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:35:15.864141 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:35:15.864141 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:35:15.864141 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 09:35:15.864141 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 09:35:15.864141 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 09:35:15.864141 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 09:35:15.864141 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:35:15.864141 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:35:15.864141 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:35:15.864141 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:35:15.864141 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:35:15.864141 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:35:15.865058 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:35:15.865058 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:35:15.865058 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:35:15.865058 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:35:15.865058 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:35:15.865058 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:35:15.865058 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:35:15.865058 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:35:15.865058 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:35:15.865058 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:35:15.865058 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 09:35:15.865058 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:35:15.865058 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 28 09:35:15.865058 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:35:15.865058 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 635/1290] 10.222.0.25 () {56 vars in 1117 bytes} [Fri Oct 28 09:35:15 2022] GET /identity/v3/services/placement => generated 90 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Oct 28 09:35:15.869847 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a9fa72-7432-46b7-8815-fc2e5bafb29f None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:15.969936 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a9fa72-7432-46b7-8815-fc2e5bafb29f None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:16.005125 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e7f424-0731-4f39-8355-0ce4cc7e66f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:16.005729 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e7f424-0731-4f39-8355-0ce4cc7e66f6 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:16.006008 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e7f424-0731-4f39-8355-0ce4cc7e66f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:16.006280 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e7f424-0731-4f39-8355-0ce4cc7e66f6 None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:16.006934 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e7f424-0731-4f39-8355-0ce4cc7e66f6 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:35:16.007236 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e7f424-0731-4f39-8355-0ce4cc7e66f6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:35:16.008582 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e7f424-0731-4f39-8355-0ce4cc7e66f6 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:35:16.013435 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 656/1291] 10.222.0.25 () {56 vars in 1116 bytes} [Fri Oct 28 09:35:15 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:35:16.020417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c32525c5-26af-4519-9700-b56817888859 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:16.148576 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c32525c5-26af-4519-9700-b56817888859 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:16.193276 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a24bf9a-0a55-4d1e-bf1a-684cd6bbf2df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:16.194007 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a24bf9a-0a55-4d1e-bf1a-684cd6bbf2df None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:16.194478 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a24bf9a-0a55-4d1e-bf1a-684cd6bbf2df None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:16.194776 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a24bf9a-0a55-4d1e-bf1a-684cd6bbf2df None admin] PATH_INFO: `/v3/services` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:16.195629 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a24bf9a-0a55-4d1e-bf1a-684cd6bbf2df None admin] RBAC: Adding query filter params (type=placement) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:35:16.195937 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a24bf9a-0a55-4d1e-bf1a-684cd6bbf2df None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:35:16.197752 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a24bf9a-0a55-4d1e-bf1a-684cd6bbf2df None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:35:16.204298 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 636/1292] 10.222.0.25 () {56 vars in 1116 bytes} [Fri Oct 28 09:35:16 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:35:18.413991 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7330f70-4db6-4156-8425-8fa341bc3f2d None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:18.414623 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7330f70-4db6-4156-8425-8fa341bc3f2d None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:18.414623 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7330f70-4db6-4156-8425-8fa341bc3f2d None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:18.415929 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 657/1293] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:35:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 09:35:18.421940 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4f7362-b151-4e74-be39-88b17dfcdc72 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:18.422521 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4f7362-b151-4e74-be39-88b17dfcdc72 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:18.422636 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c4f7362-b151-4e74-be39-88b17dfcdc72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:18.441854 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-7c4f7362-b151-4e74-be39-88b17dfcdc72 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:35:18.443236 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-7c4f7362-b151-4e74-be39-88b17dfcdc72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:35:18.475409 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:35:18.516688 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 637/1294] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:35:18 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:35:18.795254 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7330f70-4db6-4156-8425-8fa341bc3f2d None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:18.834245 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:35:18.926232 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7330f70-4db6-4156-8425-8fa341bc3f2d None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:18.966537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2947795-5dcc-45d8-aa48-ec4923ff66d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:18.967226 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2947795-5dcc-45d8-aa48-ec4923ff66d4 None admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:18.967552 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2947795-5dcc-45d8-aa48-ec4923ff66d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:18.967874 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2947795-5dcc-45d8-aa48-ec4923ff66d4 None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:18.968610 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2947795-5dcc-45d8-aa48-ec4923ff66d4 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:35:18.970100 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2947795-5dcc-45d8-aa48-ec4923ff66d4 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:35:18.988733 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 658/1295] 10.222.0.25 () {60 vars in 1141 bytes} [Fri Oct 28 09:35:18 2022] POST /identity/v3/services => generated 246 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:35:21.225469 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a68d04-5ed9-455c-9983-20844c62a99a None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:21.226052 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a68d04-5ed9-455c-9983-20844c62a99a None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:21.226052 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a68d04-5ed9-455c-9983-20844c62a99a None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:21.227385 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 638/1296] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:35:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:35:21.233073 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b85259-98dd-477f-8ff1-38289a95340f None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:21.233553 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b85259-98dd-477f-8ff1-38289a95340f None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:21.233724 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b85259-98dd-477f-8ff1-38289a95340f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:21.253903 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-54b85259-98dd-477f-8ff1-38289a95340f None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:35:21.255256 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-54b85259-98dd-477f-8ff1-38289a95340f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:35:21.344461 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 659/1297] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:35:21 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Oct 28 09:35:21.616285 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a68d04-5ed9-455c-9983-20844c62a99a None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:21.758548 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a68d04-5ed9-455c-9983-20844c62a99a None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:21.819927 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1cdb3fd-8d70-4946-93ef-0970abed7895 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:21.821001 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1cdb3fd-8d70-4946-93ef-0970abed7895 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:21.821410 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1cdb3fd-8d70-4946-93ef-0970abed7895 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:21.821856 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1cdb3fd-8d70-4946-93ef-0970abed7895 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:21.822924 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1cdb3fd-8d70-4946-93ef-0970abed7895 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:35:21.825398 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1cdb3fd-8d70-4946-93ef-0970abed7895 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:35:21.842607 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-e1cdb3fd-8d70-4946-93ef-0970abed7895 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 28 09:35:21.842607 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:35:21.842607 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 09:35:21.842607 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 09:35:21.842607 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 09:35:21.842607 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 09:35:21.842607 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 09:35:21.842607 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:35:21.842607 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 28 09:35:21.842607 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:35:21.842607 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 09:35:21.842607 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:35:21.842607 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:35:21.842607 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:35:21.842607 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:35:21.842607 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:35:21.842607 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:35:21.843887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:35:21.843887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:35:21.843887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:35:21.843887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:35:21.843887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:35:21.843887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:35:21.843887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 09:35:21.843887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 09:35:21.843887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 09:35:21.843887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 09:35:21.843887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:35:21.843887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:35:21.843887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:35:21.843887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:35:21.843887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:35:21.843887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:35:21.844937 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:35:21.844937 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:35:21.844937 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:35:21.844937 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:35:21.844937 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:35:21.844937 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:35:21.844937 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:35:21.844937 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:35:21.844937 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:35:21.844937 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:35:21.844937 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 09:35:21.844937 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:35:21.844937 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 28 09:35:21.844937 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:35:21.844937 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 639/1298] 10.222.0.25 () {56 vars in 1117 bytes} [Fri Oct 28 09:35:21 2022] GET /identity/v3/services/placement => generated 90 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:35:21.854537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b85259-98dd-477f-8ff1-38289a95340f None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:21.884714 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:35:22.020652 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b85259-98dd-477f-8ff1-38289a95340f None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:22.078879 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4d19bc-505f-4abc-a36a-45556f84340d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:22.079887 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4d19bc-505f-4abc-a36a-45556f84340d None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:22.080344 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4d19bc-505f-4abc-a36a-45556f84340d None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:22.080811 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4d19bc-505f-4abc-a36a-45556f84340d None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:22.081950 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a4d19bc-505f-4abc-a36a-45556f84340d None admin] RBAC: Adding query filter params (name=placement) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:35:22.082518 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a4d19bc-505f-4abc-a36a-45556f84340d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:35:22.085032 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a4d19bc-505f-4abc-a36a-45556f84340d None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:35:22.094354 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 660/1299] 10.222.0.25 () {56 vars in 1116 bytes} [Fri Oct 28 09:35:21 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:35:22.100834 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1cdb3fd-8d70-4946-93ef-0970abed7895 None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:22.140500 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:35:22.256267 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1cdb3fd-8d70-4946-93ef-0970abed7895 None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:22.313512 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d07fef1-1eb1-4b81-ab5f-a3068b0beea7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:22.314126 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d07fef1-1eb1-4b81-ab5f-a3068b0beea7 None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:22.314428 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d07fef1-1eb1-4b81-ab5f-a3068b0beea7 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:22.314717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d07fef1-1eb1-4b81-ab5f-a3068b0beea7 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:22.315420 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d07fef1-1eb1-4b81-ab5f-a3068b0beea7 None admin] RBAC: Adding query filter params (interface=public, service_id=379f696fe99a4f0fa14e13421c9ec0d2, region_id=RegionOne) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:35:22.315752 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d07fef1-1eb1-4b81-ab5f-a3068b0beea7 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:35:22.317149 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d07fef1-1eb1-4b81-ab5f-a3068b0beea7 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:35:22.328408 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 640/1300] 10.222.0.25 () {56 vars in 1251 bytes} [Fri Oct 28 09:35:22 2022] GET /identity/v3/endpoints?service_id=379f696fe99a4f0fa14e13421c9ec0d2&interface=public®ion_id=RegionOne => generated 194 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:35:22.332248 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4d19bc-505f-4abc-a36a-45556f84340d None admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:22.454498 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:35:22.476552 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4d19bc-505f-4abc-a36a-45556f84340d None admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:22.513770 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c6be5d-b860-4896-bbc1-ffe44e9a8a62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:22.514386 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c6be5d-b860-4896-bbc1-ffe44e9a8a62 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:22.514684 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c6be5d-b860-4896-bbc1-ffe44e9a8a62 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:22.514953 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c6be5d-b860-4896-bbc1-ffe44e9a8a62 None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:22.515557 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1c6be5d-b860-4896-bbc1-ffe44e9a8a62 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:35:22.516898 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1c6be5d-b860-4896-bbc1-ffe44e9a8a62 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:35:22.525265 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 661/1301] 10.222.0.25 () {56 vars in 1087 bytes} [Fri Oct 28 09:35:22 2022] GET /identity/v3/services => generated 2191 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:35:24.806299 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f42472-1d90-4f48-92d4-12195a423d82 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:24.806901 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f42472-1d90-4f48-92d4-12195a423d82 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:24.806901 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f42472-1d90-4f48-92d4-12195a423d82 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:24.808729 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 641/1302] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:35:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:35:24.814058 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3a9a04-9609-425f-a6ec-2124773ed05b None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:24.814601 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3a9a04-9609-425f-a6ec-2124773ed05b None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:24.814693 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3a9a04-9609-425f-a6ec-2124773ed05b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:24.833723 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-4c3a9a04-9609-425f-a6ec-2124773ed05b None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:35:24.834972 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4c3a9a04-9609-425f-a6ec-2124773ed05b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:35:24.903065 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 662/1303] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:35:24 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Oct 28 09:35:25.177285 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f42472-1d90-4f48-92d4-12195a423d82 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:25.280162 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:35:25.288796 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f42472-1d90-4f48-92d4-12195a423d82 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:25.325411 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7174e922-b54a-4316-8358-ef117511390c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:25.326047 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7174e922-b54a-4316-8358-ef117511390c None admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:25.326326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7174e922-b54a-4316-8358-ef117511390c None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:25.326628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7174e922-b54a-4316-8358-ef117511390c None admin] PATH_INFO: `/v3/services/placement` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:25.327216 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7174e922-b54a-4316-8358-ef117511390c None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:35:25.328585 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7174e922-b54a-4316-8358-ef117511390c None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:35:25.338356 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application [None req-7174e922-b54a-4316-8358-ef117511390c None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Oct 28 09:35:25.338356 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:35:25.338356 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Oct 28 09:35:25.338356 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Oct 28 09:35:25.338356 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Oct 28 09:35:25.338356 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Oct 28 09:35:25.338356 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Oct 28 09:35:25.338356 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:35:25.338356 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 28 09:35:25.338356 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:35:25.338356 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Oct 28 09:35:25.338356 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:35:25.338356 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Traceback (most recent call last): Oct 28 09:35:25.338356 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Oct 28 09:35:25.338356 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application rv = self.dispatch_request() Oct 28 09:35:25.338356 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Oct 28 09:35:25.338356 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Oct 28 09:35:25.340403 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Oct 28 09:35:25.340403 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Oct 28 09:35:25.340403 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Oct 28 09:35:25.340403 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Oct 28 09:35:25.340403 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Oct 28 09:35:25.340403 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Oct 28 09:35:25.340403 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Oct 28 09:35:25.340403 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._get_service(service_id) Oct 28 09:35:25.340403 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Oct 28 09:35:25.340403 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Oct 28 09:35:25.340403 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Oct 28 09:35:25.340403 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Oct 28 09:35:25.340403 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Oct 28 09:35:25.340403 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Oct 28 09:35:25.340403 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Oct 28 09:35:25.340403 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.get_or_create( Oct 28 09:35:25.342117 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Oct 28 09:35:25.342117 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application with Lock( Oct 28 09:35:25.342117 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Oct 28 09:35:25.342117 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self._enter() Oct 28 09:35:25.342117 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Oct 28 09:35:25.342117 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Oct 28 09:35:25.342117 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Oct 28 09:35:25.342117 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application return self.creator() Oct 28 09:35:25.342117 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Oct 28 09:35:25.342117 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application created_value = creator( Oct 28 09:35:25.342117 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Oct 28 09:35:25.342117 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Oct 28 09:35:25.342117 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Oct 28 09:35:25.342117 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: ERROR keystone.server.flask.application Oct 28 09:35:25.342117 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 642/1304] 10.222.0.25 () {56 vars in 1117 bytes} [Fri Oct 28 09:35:25 2022] GET /identity/v3/services/placement => generated 90 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Oct 28 09:35:25.347879 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3a9a04-9609-425f-a6ec-2124773ed05b None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:25.431163 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:35:25.465016 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3a9a04-9609-425f-a6ec-2124773ed05b None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:25.502983 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95eee0f-f53a-4b1d-86e2-39170f41e4b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:25.504197 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95eee0f-f53a-4b1d-86e2-39170f41e4b1 None admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:25.504482 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95eee0f-f53a-4b1d-86e2-39170f41e4b1 None admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:25.504758 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95eee0f-f53a-4b1d-86e2-39170f41e4b1 None admin] PATH_INFO: `/v3/services` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:25.505424 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d95eee0f-f53a-4b1d-86e2-39170f41e4b1 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:35:25.505731 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d95eee0f-f53a-4b1d-86e2-39170f41e4b1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:35:25.507194 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d95eee0f-f53a-4b1d-86e2-39170f41e4b1 None admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:35:25.513401 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 663/1305] 10.222.0.25 () {56 vars in 1116 bytes} [Fri Oct 28 09:35:25 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:35:25.520007 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7174e922-b54a-4316-8358-ef117511390c None admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:35:25.676972 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7174e922-b54a-4316-8358-ef117511390c None admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:35:25.732930 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff10c190-4e95-4533-9314-4c48d0d484a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:35:25.733878 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff10c190-4e95-4533-9314-4c48d0d484a3 None admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:35:25.734377 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff10c190-4e95-4533-9314-4c48d0d484a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:35:25.734922 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff10c190-4e95-4533-9314-4c48d0d484a3 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:35:25.736185 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff10c190-4e95-4533-9314-4c48d0d484a3 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:35:25.739321 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff10c190-4e95-4533-9314-4c48d0d484a3 None admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:35:25.763386 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 643/1306] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:35:25 2022] POST /identity/v3/endpoints => generated 331 bytes in 244 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:35:35.509789 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=30 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:35:35.758778 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:36:36.091406 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1d86db-9bb8-46e0-bc80-2ca4ac7cf94b None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:36:36.091990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1d86db-9bb8-46e0-bc80-2ca4ac7cf94b None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:36:36.091990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1d86db-9bb8-46e0-bc80-2ca4ac7cf94b None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:36:36.099805 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ed1d2e-fcf5-4426-b917-1b2aa7e3e7cb None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:36:36.100353 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ed1d2e-fcf5-4426-b917-1b2aa7e3e7cb None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:36:36.100353 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ed1d2e-fcf5-4426-b917-1b2aa7e3e7cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:36:36.102854 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 664/1307] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:36:36 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:36:36.248417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-03ed1d2e-fcf5-4426-b917-1b2aa7e3e7cb None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:36:36.249687 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-03ed1d2e-fcf5-4426-b917-1b2aa7e3e7cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:36:36.507822 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 644/1308] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:36:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 409 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:36:36.819842 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5067b69-deac-499d-918b-100a8cf286cc None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:36:36.820286 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5067b69-deac-499d-918b-100a8cf286cc None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:36:36.820700 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5067b69-deac-499d-918b-100a8cf286cc None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:36:36.822331 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 665/1309] 10.222.0.25 () {54 vars in 964 bytes} [Fri Oct 28 09:36:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:36:36.829034 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3178a8f5-011d-4520-bffe-104cea26c520 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:36:36.829366 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3178a8f5-011d-4520-bffe-104cea26c520 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:36:36.829603 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3178a8f5-011d-4520-bffe-104cea26c520 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:36:36.912954 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-3178a8f5-011d-4520-bffe-104cea26c520 None None] MFA Rules not processed for user `1db6b98e87e64584932447c82fefc3b7`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:36:36.914081 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-3178a8f5-011d-4520-bffe-104cea26c520 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:36:36.936363 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:36:37.016345 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 645/1310] 10.222.0.25 () {58 vars in 1061 bytes} [Fri Oct 28 09:36:36 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:36:37.026057 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5067b69-deac-499d-918b-100a8cf286cc None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:36:37.027959 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d5067b69-deac-499d-918b-100a8cf286cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:36:37.165020 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5067b69-deac-499d-918b-100a8cf286cc None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:36:37.259229 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-454a662a-b4eb-44ba-bc45-fdf18f96bad4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:36:37.260009 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454a662a-b4eb-44ba-bc45-fdf18f96bad4 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:36:37.260394 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454a662a-b4eb-44ba-bc45-fdf18f96bad4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:36:37.260797 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454a662a-b4eb-44ba-bc45-fdf18f96bad4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:36:37.262541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-454a662a-b4eb-44ba-bc45-fdf18f96bad4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:36:37.331380 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:36:37.405869 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-454a662a-b4eb-44ba-bc45-fdf18f96bad4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:36:37.407706 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-454a662a-b4eb-44ba-bc45-fdf18f96bad4 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:36:37.432743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 666/1311] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:36:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 407 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:36:39.918465 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37d2622-d36c-463b-bfb9-ea9da5467e15 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:36:39.919061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37d2622-d36c-463b-bfb9-ea9da5467e15 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:36:39.919061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37d2622-d36c-463b-bfb9-ea9da5467e15 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:36:39.920333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 646/1312] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:36:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:36:39.926261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa4007e-9f18-4e62-8eb8-f595d35ef6f5 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:36:39.926744 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa4007e-9f18-4e62-8eb8-f595d35ef6f5 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:36:39.926855 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa4007e-9f18-4e62-8eb8-f595d35ef6f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:36:40.012308 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-0fa4007e-9f18-4e62-8eb8-f595d35ef6f5 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:36:40.013538 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-0fa4007e-9f18-4e62-8eb8-f595d35ef6f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:36:40.114468 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 667/1313] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:36:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:36:40.398005 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37d2622-d36c-463b-bfb9-ea9da5467e15 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:36:40.399578 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-b37d2622-d36c-463b-bfb9-ea9da5467e15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:36:40.501625 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37d2622-d36c-463b-bfb9-ea9da5467e15 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:36:40.575706 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:36:40.578254 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c16324-eca1-47bd-a9ef-e432c9b4365a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:36:40.579118 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c16324-eca1-47bd-a9ef-e432c9b4365a service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:36:40.579413 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c16324-eca1-47bd-a9ef-e432c9b4365a service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:36:40.579916 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c16324-eca1-47bd-a9ef-e432c9b4365a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:36:40.582243 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-e1c16324-eca1-47bd-a9ef-e432c9b4365a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:36:40.688844 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1c16324-eca1-47bd-a9ef-e432c9b4365a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:36:40.690734 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1c16324-eca1-47bd-a9ef-e432c9b4365a service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:36:40.709791 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 647/1314] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:36:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:36:43.172333 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c99489-2ae2-4ef2-95f6-097695b26dbf None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:36:43.172899 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c99489-2ae2-4ef2-95f6-097695b26dbf None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:36:43.172899 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c99489-2ae2-4ef2-95f6-097695b26dbf None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:36:43.174269 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 668/1315] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:36:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:36:43.179693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab994d16-7755-4a75-a79b-fc24815100e4 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:36:43.180125 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab994d16-7755-4a75-a79b-fc24815100e4 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:36:43.180237 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab994d16-7755-4a75-a79b-fc24815100e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:36:43.244606 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-ab994d16-7755-4a75-a79b-fc24815100e4 None None] MFA Rules not processed for user `c717178de72f493788554ab410b527de`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:36:43.245807 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-ab994d16-7755-4a75-a79b-fc24815100e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:36:43.297508 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:36:43.342255 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 648/1316] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:36:43 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:36:43.642173 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff77f897-fc56-4982-aa34-0ed4fa0086a9 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:36:43.642493 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff77f897-fc56-4982-aa34-0ed4fa0086a9 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:36:43.642771 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff77f897-fc56-4982-aa34-0ed4fa0086a9 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:36:43.644128 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 669/1317] 10.222.0.25 () {54 vars in 964 bytes} [Fri Oct 28 09:36:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:36:43.650424 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb984284-56b4-4dc1-8c5e-16f9738862cc None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:36:43.650713 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb984284-56b4-4dc1-8c5e-16f9738862cc None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:36:43.650992 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb984284-56b4-4dc1-8c5e-16f9738862cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:36:43.729357 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-bb984284-56b4-4dc1-8c5e-16f9738862cc None None] MFA Rules not processed for user `1db6b98e87e64584932447c82fefc3b7`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:36:43.730590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-bb984284-56b4-4dc1-8c5e-16f9738862cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:36:43.807781 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 649/1318] 10.222.0.25 () {58 vars in 1061 bytes} [Fri Oct 28 09:36:43 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:36:43.817388 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff77f897-fc56-4982-aa34-0ed4fa0086a9 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:36:43.818982 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ff77f897-fc56-4982-aa34-0ed4fa0086a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:36:43.829917 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:36:43.925620 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff77f897-fc56-4982-aa34-0ed4fa0086a9 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:36:43.999103 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec092fef-cda9-491b-ba39-8c7b62b99ab3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:36:43.999715 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec092fef-cda9-491b-ba39-8c7b62b99ab3 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:36:43.999998 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec092fef-cda9-491b-ba39-8c7b62b99ab3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:36:44.000271 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec092fef-cda9-491b-ba39-8c7b62b99ab3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:36:44.001856 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ec092fef-cda9-491b-ba39-8c7b62b99ab3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:36:44.063243 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:36:44.110765 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec092fef-cda9-491b-ba39-8c7b62b99ab3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:36:44.112179 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec092fef-cda9-491b-ba39-8c7b62b99ab3 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:36:44.130277 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 670/1319] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:36:43 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:36:47.216760 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef62b318-7b94-4a7c-a51a-331cd16b31b4 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:36:47.217321 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef62b318-7b94-4a7c-a51a-331cd16b31b4 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:36:47.217321 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef62b318-7b94-4a7c-a51a-331cd16b31b4 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:36:47.218688 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 650/1320] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:36:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:36:47.223945 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5902dc-f07b-4d9c-9e55-c9665e4989ba None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:36:47.224351 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5902dc-f07b-4d9c-9e55-c9665e4989ba None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:36:47.224493 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5902dc-f07b-4d9c-9e55-c9665e4989ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:36:47.296942 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-8b5902dc-f07b-4d9c-9e55-c9665e4989ba None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:36:47.298159 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-8b5902dc-f07b-4d9c-9e55-c9665e4989ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:36:47.395765 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 671/1321] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:36:47 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:36:47.681209 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef62b318-7b94-4a7c-a51a-331cd16b31b4 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:36:47.682778 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-ef62b318-7b94-4a7c-a51a-331cd16b31b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:36:47.733420 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:36:47.788942 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef62b318-7b94-4a7c-a51a-331cd16b31b4 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:36:47.861945 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46028db-777f-40f9-bdac-a83393688ce2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:36:47.862675 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46028db-777f-40f9-bdac-a83393688ce2 service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:36:47.862877 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46028db-777f-40f9-bdac-a83393688ce2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:36:47.863231 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46028db-777f-40f9-bdac-a83393688ce2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:36:47.864762 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-c46028db-777f-40f9-bdac-a83393688ce2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:36:47.959973 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:36:47.973556 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c46028db-777f-40f9-bdac-a83393688ce2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:36:47.975030 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c46028db-777f-40f9-bdac-a83393688ce2 service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:36:47.993308 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 651/1322] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:36:47 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:36:50.920638 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c306c98-8a1e-4375-b74f-f324299f33e0 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:36:50.921202 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c306c98-8a1e-4375-b74f-f324299f33e0 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:36:50.921202 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c306c98-8a1e-4375-b74f-f324299f33e0 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:36:50.922719 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 672/1323] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:36:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:36:50.928492 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c15a6fc-6ffb-44f1-bab2-9135491bcb7c None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:36:50.928933 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c15a6fc-6ffb-44f1-bab2-9135491bcb7c None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:36:50.929211 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c15a6fc-6ffb-44f1-bab2-9135491bcb7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:36:51.004823 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-1c15a6fc-6ffb-44f1-bab2-9135491bcb7c None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:36:51.006068 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-1c15a6fc-6ffb-44f1-bab2-9135491bcb7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:36:51.119964 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 652/1324] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:36:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:36:51.406571 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c306c98-8a1e-4375-b74f-f324299f33e0 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:36:51.408090 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-8c306c98-8a1e-4375-b74f-f324299f33e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:36:51.429964 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:36:51.517426 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c306c98-8a1e-4375-b74f-f324299f33e0 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:36:51.604122 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f82e84-31a7-4e11-9c60-5556d9c33892 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:36:51.604731 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f82e84-31a7-4e11-9c60-5556d9c33892 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:36:51.605013 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f82e84-31a7-4e11-9c60-5556d9c33892 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:36:51.605288 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f82e84-31a7-4e11-9c60-5556d9c33892 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:36:51.606912 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-70f82e84-31a7-4e11-9c60-5556d9c33892 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:36:51.681501 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:36:51.723920 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70f82e84-31a7-4e11-9c60-5556d9c33892 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:36:51.725352 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70f82e84-31a7-4e11-9c60-5556d9c33892 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:36:51.745456 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 673/1325] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:36:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:36:53.978782 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113ddd49-bfac-42c6-9785-fbfb04296376 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:36:53.979320 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113ddd49-bfac-42c6-9785-fbfb04296376 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:36:53.979412 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113ddd49-bfac-42c6-9785-fbfb04296376 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:36:53.980748 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 653/1326] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:36:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:36:53.986851 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1194ed7-d4e8-4490-906b-274dc60d77bc None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:36:53.987283 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1194ed7-d4e8-4490-906b-274dc60d77bc None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:36:53.987419 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1194ed7-d4e8-4490-906b-274dc60d77bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:36:54.055531 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-d1194ed7-d4e8-4490-906b-274dc60d77bc None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:36:54.056756 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d1194ed7-d4e8-4490-906b-274dc60d77bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:36:54.170007 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 674/1327] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:36:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:36:54.454616 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-113ddd49-bfac-42c6-9785-fbfb04296376 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:36:54.456758 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-113ddd49-bfac-42c6-9785-fbfb04296376 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:36:54.516967 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:36:54.567348 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-113ddd49-bfac-42c6-9785-fbfb04296376 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:36:54.643839 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f914e8-f3b0-440b-83c3-799c21e947e6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:36:54.644443 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f914e8-f3b0-440b-83c3-799c21e947e6 service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:36:54.644723 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f914e8-f3b0-440b-83c3-799c21e947e6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:36:54.644993 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f914e8-f3b0-440b-83c3-799c21e947e6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:36:54.646611 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-52f914e8-f3b0-440b-83c3-799c21e947e6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:36:54.761101 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:36:54.763436 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f914e8-f3b0-440b-83c3-799c21e947e6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:36:54.764903 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f914e8-f3b0-440b-83c3-799c21e947e6 service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:36:54.784693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 654/1328] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:36:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:36:57.085782 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1c09c6-7bb6-486a-af17-5dbdf7b50bec None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:36:57.086536 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1c09c6-7bb6-486a-af17-5dbdf7b50bec None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:36:57.086536 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1c09c6-7bb6-486a-af17-5dbdf7b50bec None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:36:57.087853 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 675/1329] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:36:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:36:57.093315 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fb2b21-57b5-4a86-90bb-36ad38107947 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:36:57.093747 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fb2b21-57b5-4a86-90bb-36ad38107947 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:36:57.093905 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fb2b21-57b5-4a86-90bb-36ad38107947 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:36:57.160216 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-c5fb2b21-57b5-4a86-90bb-36ad38107947 None None] MFA Rules not processed for user `c717178de72f493788554ab410b527de`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:36:57.161413 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-c5fb2b21-57b5-4a86-90bb-36ad38107947 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:36:57.257631 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 655/1330] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:36:57 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:36:57.541155 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1c09c6-7bb6-486a-af17-5dbdf7b50bec None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:36:57.542755 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7c1c09c6-7bb6-486a-af17-5dbdf7b50bec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:36:57.577169 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:36:57.655076 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1c09c6-7bb6-486a-af17-5dbdf7b50bec None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:36:57.730871 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1305fecc-f431-4ed7-96e7-40aafbdbfe7a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:36:57.731607 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1305fecc-f431-4ed7-96e7-40aafbdbfe7a service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:36:57.731894 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1305fecc-f431-4ed7-96e7-40aafbdbfe7a service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:36:57.732170 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1305fecc-f431-4ed7-96e7-40aafbdbfe7a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:36:57.733758 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-1305fecc-f431-4ed7-96e7-40aafbdbfe7a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:36:57.813438 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:36:57.845192 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1305fecc-f431-4ed7-96e7-40aafbdbfe7a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:36:57.847152 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1305fecc-f431-4ed7-96e7-40aafbdbfe7a service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:36:57.864866 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 676/1331] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:36:57 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:37:00.345143 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b3a8f9-596f-4469-ac59-0d7395f67daf None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:00.346216 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b3a8f9-596f-4469-ac59-0d7395f67daf None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:00.346344 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b3a8f9-596f-4469-ac59-0d7395f67daf None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:00.347629 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 656/1332] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:37:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:37:00.353318 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5a5e3e-0bbd-4052-be10-ddff409ea3d3 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:00.353802 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5a5e3e-0bbd-4052-be10-ddff409ea3d3 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:00.353916 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5a5e3e-0bbd-4052-be10-ddff409ea3d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:00.424797 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-4f5a5e3e-0bbd-4052-be10-ddff409ea3d3 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:37:00.426348 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4f5a5e3e-0bbd-4052-be10-ddff409ea3d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:00.520843 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 677/1333] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:37:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:37:00.806149 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b3a8f9-596f-4469-ac59-0d7395f67daf None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:37:00.807716 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-d2b3a8f9-596f-4469-ac59-0d7395f67daf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:00.880531 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:37:00.915017 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b3a8f9-596f-4469-ac59-0d7395f67daf None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:37:00.988959 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82172caf-ba03-4fe3-bb97-abe9f3f83a75 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:37:00.989564 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82172caf-ba03-4fe3-bb97-abe9f3f83a75 service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:00.989848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82172caf-ba03-4fe3-bb97-abe9f3f83a75 service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:00.990126 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82172caf-ba03-4fe3-bb97-abe9f3f83a75 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:00.991768 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-82172caf-ba03-4fe3-bb97-abe9f3f83a75 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:01.109491 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82172caf-ba03-4fe3-bb97-abe9f3f83a75 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:37:01.111080 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82172caf-ba03-4fe3-bb97-abe9f3f83a75 service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:37:01.122865 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:37:01.133811 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 657/1334] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:37:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:37:03.345179 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abbad73-0fcf-4e36-9a00-d5ee488d6cc0 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:03.346248 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abbad73-0fcf-4e36-9a00-d5ee488d6cc0 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:03.346344 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abbad73-0fcf-4e36-9a00-d5ee488d6cc0 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:03.347680 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 678/1335] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:37:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:37:03.352985 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcc9a67-4e31-4527-8162-125f72d1db52 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:03.353376 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcc9a67-4e31-4527-8162-125f72d1db52 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:03.353535 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcc9a67-4e31-4527-8162-125f72d1db52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:03.418941 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-bfcc9a67-4e31-4527-8162-125f72d1db52 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:37:03.420147 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-bfcc9a67-4e31-4527-8162-125f72d1db52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:03.513720 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 658/1336] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:37:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:37:03.797838 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8abbad73-0fcf-4e36-9a00-d5ee488d6cc0 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:37:03.800023 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-8abbad73-0fcf-4e36-9a00-d5ee488d6cc0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:03.842306 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:37:03.911649 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8abbad73-0fcf-4e36-9a00-d5ee488d6cc0 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:37:03.992734 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e19c1429-aa12-4170-80c8-d53e343fd6b1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:37:03.993352 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19c1429-aa12-4170-80c8-d53e343fd6b1 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:03.993630 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19c1429-aa12-4170-80c8-d53e343fd6b1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:03.993898 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19c1429-aa12-4170-80c8-d53e343fd6b1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:03.995563 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e19c1429-aa12-4170-80c8-d53e343fd6b1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:04.085994 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:37:04.113490 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e19c1429-aa12-4170-80c8-d53e343fd6b1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:37:04.116463 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e19c1429-aa12-4170-80c8-d53e343fd6b1 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:37:04.135026 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 679/1337] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:37:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:37:06.628911 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74eaa608-0501-442b-9b91-71751e7a78b0 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:06.629429 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74eaa608-0501-442b-9b91-71751e7a78b0 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:06.629540 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74eaa608-0501-442b-9b91-71751e7a78b0 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:06.631039 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 659/1338] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:37:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:37:06.637226 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec04802-7b62-4186-9390-6d11a189dc9f None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:06.637754 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec04802-7b62-4186-9390-6d11a189dc9f None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:06.637754 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec04802-7b62-4186-9390-6d11a189dc9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:06.707385 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-4ec04802-7b62-4186-9390-6d11a189dc9f None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:37:06.708633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4ec04802-7b62-4186-9390-6d11a189dc9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:06.806489 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 680/1339] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:37:06 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:37:07.088415 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74eaa608-0501-442b-9b91-71751e7a78b0 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:37:07.089974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-74eaa608-0501-442b-9b91-71751e7a78b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:07.176186 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:37:07.204517 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74eaa608-0501-442b-9b91-71751e7a78b0 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:37:07.279557 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7ff81c2-c3ef-4251-a007-740cbe6dcfae service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:37:07.280167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ff81c2-c3ef-4251-a007-740cbe6dcfae service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:07.280447 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ff81c2-c3ef-4251-a007-740cbe6dcfae service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:07.280719 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7ff81c2-c3ef-4251-a007-740cbe6dcfae service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:07.282319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-b7ff81c2-c3ef-4251-a007-740cbe6dcfae service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:07.390355 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7ff81c2-c3ef-4251-a007-740cbe6dcfae service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:37:07.391747 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7ff81c2-c3ef-4251-a007-740cbe6dcfae service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:37:07.410223 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 660/1340] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:37:07 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:37:11.546070 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca0cbfe-62f7-4bbb-b145-b1139d21efc4 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:11.546716 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca0cbfe-62f7-4bbb-b145-b1139d21efc4 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:11.546716 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca0cbfe-62f7-4bbb-b145-b1139d21efc4 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:11.548010 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 681/1341] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:37:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:37:11.553516 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2776a42f-3a8c-4e45-89cb-e4719f67ea1a None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:11.554005 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2776a42f-3a8c-4e45-89cb-e4719f67ea1a None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:11.554096 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2776a42f-3a8c-4e45-89cb-e4719f67ea1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:11.559633 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:37:11.621630 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-2776a42f-3a8c-4e45-89cb-e4719f67ea1a None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:37:11.622782 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-2776a42f-3a8c-4e45-89cb-e4719f67ea1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:11.722207 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 661/1342] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:37:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:37:12.003850 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ca0cbfe-62f7-4bbb-b145-b1139d21efc4 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:37:12.005371 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4ca0cbfe-62f7-4bbb-b145-b1139d21efc4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:12.054636 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:37:12.111715 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ca0cbfe-62f7-4bbb-b145-b1139d21efc4 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:37:12.190360 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d9e23b-d410-4bc3-b703-ac09f29247f5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:37:12.190992 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d9e23b-d410-4bc3-b703-ac09f29247f5 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:12.191270 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d9e23b-d410-4bc3-b703-ac09f29247f5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:12.191541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d9e23b-d410-4bc3-b703-ac09f29247f5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:12.193121 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-17d9e23b-d410-4bc3-b703-ac09f29247f5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:12.286750 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:37:12.304467 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d9e23b-d410-4bc3-b703-ac09f29247f5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:37:12.305886 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d9e23b-d410-4bc3-b703-ac09f29247f5 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:37:12.324625 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 682/1343] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:37:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:37:15.097464 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-becf0cf2-f274-4d66-b811-b33ff0d875a3 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:15.098048 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-becf0cf2-f274-4d66-b811-b33ff0d875a3 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:15.098048 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-becf0cf2-f274-4d66-b811-b33ff0d875a3 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:15.099391 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 662/1344] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:37:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:37:15.105346 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c16910-4376-4de5-9e7c-1eb991ede905 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:15.105865 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c16910-4376-4de5-9e7c-1eb991ede905 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:15.105987 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59c16910-4376-4de5-9e7c-1eb991ede905 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:15.183659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-59c16910-4376-4de5-9e7c-1eb991ede905 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:37:15.184893 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-59c16910-4376-4de5-9e7c-1eb991ede905 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:15.287475 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 683/1345] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:37:15 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:37:15.570372 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-becf0cf2-f274-4d66-b811-b33ff0d875a3 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:37:15.572279 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-becf0cf2-f274-4d66-b811-b33ff0d875a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:15.670763 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:37:15.689279 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-becf0cf2-f274-4d66-b811-b33ff0d875a3 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:37:15.771087 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d89df97-a5f2-4d41-9e40-9dbd31e3a7dc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:37:15.771712 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d89df97-a5f2-4d41-9e40-9dbd31e3a7dc service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:15.772276 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d89df97-a5f2-4d41-9e40-9dbd31e3a7dc service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:15.772276 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d89df97-a5f2-4d41-9e40-9dbd31e3a7dc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:15.774008 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-0d89df97-a5f2-4d41-9e40-9dbd31e3a7dc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:15.900877 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d89df97-a5f2-4d41-9e40-9dbd31e3a7dc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:37:15.902316 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d89df97-a5f2-4d41-9e40-9dbd31e3a7dc service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:37:15.926754 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 663/1346] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:37:15 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 357 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:37:20.376274 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7810c34f-e6d3-40fb-8415-d60cddda1787 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:20.377569 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7810c34f-e6d3-40fb-8415-d60cddda1787 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:20.377569 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7810c34f-e6d3-40fb-8415-d60cddda1787 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:20.378925 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 684/1347] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:37:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 09:37:20.384864 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda7fdc6-2c6c-48f1-9df6-531cc637c3fb None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:20.385425 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda7fdc6-2c6c-48f1-9df6-531cc637c3fb None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:20.385634 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda7fdc6-2c6c-48f1-9df6-531cc637c3fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:20.406903 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:37:20.478821 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-bda7fdc6-2c6c-48f1-9df6-531cc637c3fb None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:37:20.480250 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-bda7fdc6-2c6c-48f1-9df6-531cc637c3fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:20.640540 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 664/1348] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:37:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 256 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:37:20.931121 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7810c34f-e6d3-40fb-8415-d60cddda1787 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:37:20.932692 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7810c34f-e6d3-40fb-8415-d60cddda1787 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:20.995922 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:37:21.043521 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7810c34f-e6d3-40fb-8415-d60cddda1787 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:37:21.128182 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7699baf-8f0b-4671-bd9f-783321129b92 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:37:21.129055 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7699baf-8f0b-4671-bd9f-783321129b92 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:21.129533 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7699baf-8f0b-4671-bd9f-783321129b92 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:21.129919 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7699baf-8f0b-4671-bd9f-783321129b92 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:21.131842 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a7699baf-8f0b-4671-bd9f-783321129b92 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:21.241820 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:37:21.246016 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7699baf-8f0b-4671-bd9f-783321129b92 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:37:21.247495 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7699baf-8f0b-4671-bd9f-783321129b92 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:37:21.266158 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 685/1349] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:37:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:37:26.372944 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07b6df3-4c9b-420b-86ea-78db74131773 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:26.373511 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07b6df3-4c9b-420b-86ea-78db74131773 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:26.373511 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b07b6df3-4c9b-420b-86ea-78db74131773 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:26.375237 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 665/1350] 10.222.0.25 () {54 vars in 921 bytes} [Fri Oct 28 09:37:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:37:26.381662 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347f839d-3273-418d-aebf-0d3d2aefa2dc None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:26.383577 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347f839d-3273-418d-aebf-0d3d2aefa2dc None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:26.383577 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347f839d-3273-418d-aebf-0d3d2aefa2dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:26.491639 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-347f839d-3273-418d-aebf-0d3d2aefa2dc None None] MFA Rules not processed for user `3f1d95049d2d4593bf52670b1a0035fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:37:26.493147 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-347f839d-3273-418d-aebf-0d3d2aefa2dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:26.637368 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 686/1351] 10.222.0.25 () {58 vars in 1018 bytes} [Fri Oct 28 09:37:26 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 256 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:37:30.637303 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=59 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:37:31.201815 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306cbb80-b0f7-4d06-afe8-2ada072c3871 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:31.202181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306cbb80-b0f7-4d06-afe8-2ada072c3871 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:31.202700 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306cbb80-b0f7-4d06-afe8-2ada072c3871 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:31.204273 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 666/1352] 10.222.0.25 () {54 vars in 921 bytes} [Fri Oct 28 09:37:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:37:31.210346 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509a6a11-2f14-48c3-8afe-4787395c4ff3 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:31.210848 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509a6a11-2f14-48c3-8afe-4787395c4ff3 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:31.211094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509a6a11-2f14-48c3-8afe-4787395c4ff3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:31.269797 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:37:31.294667 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-509a6a11-2f14-48c3-8afe-4787395c4ff3 None None] MFA Rules not processed for user `3f1d95049d2d4593bf52670b1a0035fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:37:31.295921 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-509a6a11-2f14-48c3-8afe-4787395c4ff3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:31.422192 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 687/1353] 10.222.0.25 () {58 vars in 1018 bytes} [Fri Oct 28 09:37:31 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:37:32.709941 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385212cd-d48a-48ec-8a6b-1af1ef590320 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:32.710580 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385212cd-d48a-48ec-8a6b-1af1ef590320 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:32.710580 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385212cd-d48a-48ec-8a6b-1af1ef590320 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:32.711855 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 667/1354] 10.222.0.25 () {54 vars in 921 bytes} [Fri Oct 28 09:37:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:37:32.717037 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96207f46-7d2a-420a-8aa2-2bada06f4492 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:32.717464 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96207f46-7d2a-420a-8aa2-2bada06f4492 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:32.717622 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96207f46-7d2a-420a-8aa2-2bada06f4492 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:32.798687 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-96207f46-7d2a-420a-8aa2-2bada06f4492 None None] MFA Rules not processed for user `3f1d95049d2d4593bf52670b1a0035fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:37:32.799934 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-96207f46-7d2a-420a-8aa2-2bada06f4492 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:32.842797 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:37:32.913743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 688/1355] 10.222.0.25 () {58 vars in 1018 bytes} [Fri Oct 28 09:37:32 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:37:36.431606 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab22d685-5f4c-4179-aea2-424a1aec74d2 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:36.432145 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab22d685-5f4c-4179-aea2-424a1aec74d2 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:36.432145 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab22d685-5f4c-4179-aea2-424a1aec74d2 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:36.433560 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 668/1356] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:37:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:37:36.439486 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f957ecc8-02a6-4c8d-9291-9fed2559d691 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:36.439950 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f957ecc8-02a6-4c8d-9291-9fed2559d691 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:36.440055 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f957ecc8-02a6-4c8d-9291-9fed2559d691 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:36.512445 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-f957ecc8-02a6-4c8d-9291-9fed2559d691 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:37:36.513675 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-f957ecc8-02a6-4c8d-9291-9fed2559d691 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:36.622909 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 689/1357] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:37:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:37:36.934122 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b43dd47-af5d-4c6c-9ec5-bf3f7161dc64 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:36.934933 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b43dd47-af5d-4c6c-9ec5-bf3f7161dc64 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:36.935443 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b43dd47-af5d-4c6c-9ec5-bf3f7161dc64 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:36.937024 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 669/1358] 10.222.0.25 () {54 vars in 960 bytes} [Fri Oct 28 09:37:36 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:37:36.946744 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52de1ee-fe2d-40da-9603-c23b07750949 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:36.948003 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52de1ee-fe2d-40da-9603-c23b07750949 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:36.948720 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52de1ee-fe2d-40da-9603-c23b07750949 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:36.974843 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:37:37.050819 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-f52de1ee-fe2d-40da-9603-c23b07750949 None None] MFA Rules not processed for user `688ceb201d304c2da5373c3cf36c2d79`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:37:37.052600 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-f52de1ee-fe2d-40da-9603-c23b07750949 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:37.185698 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 690/1359] 10.222.0.25 () {58 vars in 1057 bytes} [Fri Oct 28 09:37:36 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 241 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:37:37.194039 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b43dd47-af5d-4c6c-9ec5-bf3f7161dc64 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:37:37.195570 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-5b43dd47-af5d-4c6c-9ec5-bf3f7161dc64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:37.357311 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b43dd47-af5d-4c6c-9ec5-bf3f7161dc64 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:37:37.436457 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:37:37.477705 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5995afa2-ac2c-4707-8308-cbe9a11ca769 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:37:37.478307 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5995afa2-ac2c-4707-8308-cbe9a11ca769 service nova] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:37.478609 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5995afa2-ac2c-4707-8308-cbe9a11ca769 service nova] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:37.478885 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5995afa2-ac2c-4707-8308-cbe9a11ca769 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:37.480451 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-5995afa2-ac2c-4707-8308-cbe9a11ca769 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:37.585399 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5995afa2-ac2c-4707-8308-cbe9a11ca769 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:37:37.587050 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5995afa2-ac2c-4707-8308-cbe9a11ca769 service nova] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:37:37.606492 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 670/1360] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:37:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 413 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:37:40.027742 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93931143-bc6f-4025-8591-c31d84124e0c None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:40.027742 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93931143-bc6f-4025-8591-c31d84124e0c None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:40.028536 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93931143-bc6f-4025-8591-c31d84124e0c None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:40.033533 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 691/1361] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:37:40 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:37:40.034801 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ceecd5-48a5-4dd7-b528-1baa91cb5df9 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:40.035341 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ceecd5-48a5-4dd7-b528-1baa91cb5df9 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:40.035341 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ceecd5-48a5-4dd7-b528-1baa91cb5df9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:40.116003 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-29ceecd5-48a5-4dd7-b528-1baa91cb5df9 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:37:40.117267 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-29ceecd5-48a5-4dd7-b528-1baa91cb5df9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:40.160814 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:37:40.229508 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 671/1362] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:37:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:37:40.537515 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93931143-bc6f-4025-8591-c31d84124e0c None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:37:40.539809 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-93931143-bc6f-4025-8591-c31d84124e0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:40.636454 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:37:40.752410 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93931143-bc6f-4025-8591-c31d84124e0c None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:37:40.887652 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae34eaf-c986-4b89-8255-7913a72dc03c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:37:40.888363 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae34eaf-c986-4b89-8255-7913a72dc03c service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:40.888730 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae34eaf-c986-4b89-8255-7913a72dc03c service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:40.889008 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae34eaf-c986-4b89-8255-7913a72dc03c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:40.890655 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-6ae34eaf-c986-4b89-8255-7913a72dc03c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:40.915187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:37:40.985877 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae34eaf-c986-4b89-8255-7913a72dc03c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:37:40.987621 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae34eaf-c986-4b89-8255-7913a72dc03c service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:37:41.006705 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 692/1363] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:37:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 470 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:37:43.528939 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5937063-da01-4965-bb60-bc89c07ceff6 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:43.529492 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5937063-da01-4965-bb60-bc89c07ceff6 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:43.529492 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5937063-da01-4965-bb60-bc89c07ceff6 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:43.530884 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 672/1364] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:37:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:37:43.536649 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d300270-84e3-4522-a854-2250c9559c63 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:43.537947 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d300270-84e3-4522-a854-2250c9559c63 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:43.538329 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d300270-84e3-4522-a854-2250c9559c63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:43.746583 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-9d300270-84e3-4522-a854-2250c9559c63 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:37:43.747821 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-9d300270-84e3-4522-a854-2250c9559c63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:43.850000 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:37:43.851879 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 693/1365] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:37:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:37:44.146965 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef46a527-9fc2-4900-b7dc-91ec1268fb03 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:44.147250 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef46a527-9fc2-4900-b7dc-91ec1268fb03 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:44.147663 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef46a527-9fc2-4900-b7dc-91ec1268fb03 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:44.149126 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 673/1366] 10.222.0.25 () {54 vars in 960 bytes} [Fri Oct 28 09:37:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:37:44.154826 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24dd0bc7-956b-45f2-b5f4-86a28b23a2c6 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:44.155696 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24dd0bc7-956b-45f2-b5f4-86a28b23a2c6 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:44.155980 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24dd0bc7-956b-45f2-b5f4-86a28b23a2c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:44.241730 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-24dd0bc7-956b-45f2-b5f4-86a28b23a2c6 None None] MFA Rules not processed for user `688ceb201d304c2da5373c3cf36c2d79`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:37:44.243003 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-24dd0bc7-956b-45f2-b5f4-86a28b23a2c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:44.375153 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 694/1367] 10.222.0.25 () {58 vars in 1057 bytes} [Fri Oct 28 09:37:44 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 221 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:37:44.384130 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef46a527-9fc2-4900-b7dc-91ec1268fb03 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:37:44.385760 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-ef46a527-9fc2-4900-b7dc-91ec1268fb03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:44.549573 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef46a527-9fc2-4900-b7dc-91ec1268fb03 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:37:44.581633 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:37:44.682683 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51061041-0cf0-47df-98a3-89a2159889be None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:44.683193 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51061041-0cf0-47df-98a3-89a2159889be None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:44.683301 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d9e271-31db-48ba-aee3-20aadfd95992 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:37:44.683406 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51061041-0cf0-47df-98a3-89a2159889be None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:44.683693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d9e271-31db-48ba-aee3-20aadfd95992 service nova] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:44.683921 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d9e271-31db-48ba-aee3-20aadfd95992 service nova] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:44.684193 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d9e271-31db-48ba-aee3-20aadfd95992 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:44.684762 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 695/1368] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:37:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:37:44.685986 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-d7d9e271-31db-48ba-aee3-20aadfd95992 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:44.690466 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05211dd3-5885-4daf-a75c-60767a5bdf30 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:44.691018 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05211dd3-5885-4daf-a75c-60767a5bdf30 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:44.691524 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05211dd3-5885-4daf-a75c-60767a5bdf30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:44.755757 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:37:44.765547 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-05211dd3-5885-4daf-a75c-60767a5bdf30 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:37:44.767323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-05211dd3-5885-4daf-a75c-60767a5bdf30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:44.787085 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7d9e271-31db-48ba-aee3-20aadfd95992 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:37:44.789098 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7d9e271-31db-48ba-aee3-20aadfd95992 service nova] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:37:44.812173 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 674/1369] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:37:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 428 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:37:44.888124 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 696/1370] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:37:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:37:45.222477 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ecdcdf-293a-45cf-88ae-bf38e7fcee4c None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:45.222477 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ecdcdf-293a-45cf-88ae-bf38e7fcee4c None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:45.222836 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ecdcdf-293a-45cf-88ae-bf38e7fcee4c None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:45.226428 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 675/1371] 10.222.0.25 () {54 vars in 957 bytes} [Fri Oct 28 09:37:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:37:45.229314 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4565cd0e-f436-4dee-9a16-0164e534f49e None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:45.229715 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4565cd0e-f436-4dee-9a16-0164e534f49e None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:45.230095 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4565cd0e-f436-4dee-9a16-0164e534f49e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:45.317243 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-4565cd0e-f436-4dee-9a16-0164e534f49e None None] MFA Rules not processed for user `1021f1cc73b54d6eaa647a5f49815ef8`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:37:45.318643 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4565cd0e-f436-4dee-9a16-0164e534f49e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:45.398155 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:37:45.414422 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ecdcdf-293a-45cf-88ae-bf38e7fcee4c None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:37:45.415938 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-51ecdcdf-293a-45cf-88ae-bf38e7fcee4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:45.418820 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 697/1372] 10.222.0.25 () {58 vars in 1054 bytes} [Fri Oct 28 09:37:45 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:37:45.445609 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:37:45.531615 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ecdcdf-293a-45cf-88ae-bf38e7fcee4c None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:37:45.637645 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315bc52e-2222-44b9-9dbf-720c230d877b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:37:45.638448 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315bc52e-2222-44b9-9dbf-720c230d877b service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:45.639409 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315bc52e-2222-44b9-9dbf-720c230d877b service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:45.639409 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315bc52e-2222-44b9-9dbf-720c230d877b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:45.641956 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-315bc52e-2222-44b9-9dbf-720c230d877b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:45.768714 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:37:45.825225 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-315bc52e-2222-44b9-9dbf-720c230d877b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:37:45.827372 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-315bc52e-2222-44b9-9dbf-720c230d877b service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:37:45.855947 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 676/1373] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:37:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 442 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:37:47.090960 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29431b81-9908-45d0-99ac-26c92b5ff353 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:47.091543 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29431b81-9908-45d0-99ac-26c92b5ff353 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:47.091543 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29431b81-9908-45d0-99ac-26c92b5ff353 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:47.092854 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 698/1374] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:37:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:37:47.098217 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9892c2-0fcc-4000-a3ae-142a678e01ac None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:47.098711 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9892c2-0fcc-4000-a3ae-142a678e01ac None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:47.098802 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9892c2-0fcc-4000-a3ae-142a678e01ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:47.168263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-cf9892c2-0fcc-4000-a3ae-142a678e01ac None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:37:47.169521 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-cf9892c2-0fcc-4000-a3ae-142a678e01ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:47.270951 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 677/1375] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:37:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:37:47.567834 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29431b81-9908-45d0-99ac-26c92b5ff353 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:37:47.569419 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-29431b81-9908-45d0-99ac-26c92b5ff353 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:47.730122 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29431b81-9908-45d0-99ac-26c92b5ff353 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:37:47.804870 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:37:47.854721 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-559a4e67-9a24-497e-a101-7df79dda7c36 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:37:47.855532 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559a4e67-9a24-497e-a101-7df79dda7c36 service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:47.855636 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559a4e67-9a24-497e-a101-7df79dda7c36 service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:47.856045 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559a4e67-9a24-497e-a101-7df79dda7c36 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:47.858166 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-559a4e67-9a24-497e-a101-7df79dda7c36 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:47.960695 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-559a4e67-9a24-497e-a101-7df79dda7c36 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:37:47.962372 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-559a4e67-9a24-497e-a101-7df79dda7c36 service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:37:47.981545 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 699/1376] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:37:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 414 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:37:50.199355 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58778b09-a653-45b2-b23b-745f666e884f None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:50.199959 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58778b09-a653-45b2-b23b-745f666e884f None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:50.200146 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58778b09-a653-45b2-b23b-745f666e884f None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:50.201767 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 678/1377] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:37:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:37:50.207627 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b14c0b-bf2d-47d3-bd4c-34aa67e95316 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:50.208079 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b14c0b-bf2d-47d3-bd4c-34aa67e95316 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:50.208160 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b14c0b-bf2d-47d3-bd4c-34aa67e95316 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:50.288187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-64b14c0b-bf2d-47d3-bd4c-34aa67e95316 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:37:50.289549 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-64b14c0b-bf2d-47d3-bd4c-34aa67e95316 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:50.321817 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:37:50.394911 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 700/1378] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:37:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:37:50.689202 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58778b09-a653-45b2-b23b-745f666e884f None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:37:50.690955 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-58778b09-a653-45b2-b23b-745f666e884f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:50.734317 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:37:50.855572 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58778b09-a653-45b2-b23b-745f666e884f None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:37:50.995106 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:37:51.001946 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a72f96c-f618-4d2f-8182-8b7c2db6ebf7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:37:51.002917 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a72f96c-f618-4d2f-8182-8b7c2db6ebf7 service nova] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:51.003634 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a72f96c-f618-4d2f-8182-8b7c2db6ebf7 service nova] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:51.004570 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a72f96c-f618-4d2f-8182-8b7c2db6ebf7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:51.006283 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-7a72f96c-f618-4d2f-8182-8b7c2db6ebf7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:51.130230 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a72f96c-f618-4d2f-8182-8b7c2db6ebf7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:37:51.132077 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a72f96c-f618-4d2f-8182-8b7c2db6ebf7 service nova] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:37:51.155988 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 679/1379] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:37:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 468 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:37:53.403220 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf37600-ae2a-432b-841f-d6b7e221abb2 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:53.404021 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf37600-ae2a-432b-841f-d6b7e221abb2 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:53.404021 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf37600-ae2a-432b-841f-d6b7e221abb2 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:53.411234 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f070769c-1990-4335-a1da-f0c771bc4e74 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:53.411234 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f070769c-1990-4335-a1da-f0c771bc4e74 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:53.412053 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f070769c-1990-4335-a1da-f0c771bc4e74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:53.416243 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 701/1380] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:37:53 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:37:53.485671 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-f070769c-1990-4335-a1da-f0c771bc4e74 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:37:53.487464 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-f070769c-1990-4335-a1da-f0c771bc4e74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:53.580706 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:37:53.621424 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 680/1381] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:37:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:37:53.906649 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf37600-ae2a-432b-841f-d6b7e221abb2 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:37:53.908737 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ddf37600-ae2a-432b-841f-d6b7e221abb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:54.083385 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf37600-ae2a-432b-841f-d6b7e221abb2 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:37:54.098895 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:37:54.223943 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab34a94-6817-4105-93fb-544b98915423 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:37:54.224449 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab34a94-6817-4105-93fb-544b98915423 service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:54.224747 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab34a94-6817-4105-93fb-544b98915423 service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:54.225159 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab34a94-6817-4105-93fb-544b98915423 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:54.226798 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-fab34a94-6817-4105-93fb-544b98915423 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:54.340390 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fab34a94-6817-4105-93fb-544b98915423 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:37:54.341976 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fab34a94-6817-4105-93fb-544b98915423 service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:37:54.361184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 702/1382] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:37:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 455 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:37:56.611592 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24d6188-74d3-4f7d-ae4e-eb9baa8cb664 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:56.612127 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24d6188-74d3-4f7d-ae4e-eb9baa8cb664 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:56.612127 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24d6188-74d3-4f7d-ae4e-eb9baa8cb664 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:56.613484 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 681/1383] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:37:56 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:37:56.621105 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c587ba85-9f01-4078-a18b-70c1d55c305b None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:56.621562 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c587ba85-9f01-4078-a18b-70c1d55c305b None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:56.621871 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c587ba85-9f01-4078-a18b-70c1d55c305b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:56.653770 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:37:56.719993 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-c587ba85-9f01-4078-a18b-70c1d55c305b None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:37:56.721735 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-c587ba85-9f01-4078-a18b-70c1d55c305b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:56.943891 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 703/1384] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:37:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 324 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:37:57.352419 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a24d6188-74d3-4f7d-ae4e-eb9baa8cb664 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:37:57.353529 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-a24d6188-74d3-4f7d-ae4e-eb9baa8cb664 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:57.529579 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a24d6188-74d3-4f7d-ae4e-eb9baa8cb664 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:37:57.601225 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:37:57.676555 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f40846-89ee-4ba9-bcae-b1ce6514bdf9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:37:57.677225 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f40846-89ee-4ba9-bcae-b1ce6514bdf9 service nova] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:37:57.677446 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f40846-89ee-4ba9-bcae-b1ce6514bdf9 service nova] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:37:57.677722 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f40846-89ee-4ba9-bcae-b1ce6514bdf9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:37:57.679373 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-b7f40846-89ee-4ba9-bcae-b1ce6514bdf9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:37:57.787883 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f40846-89ee-4ba9-bcae-b1ce6514bdf9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:37:57.789551 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f40846-89ee-4ba9-bcae-b1ce6514bdf9 service nova] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:37:57.808966 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 682/1385] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:37:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 457 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:38:00.384044 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ef74ac-db3c-4a9b-9811-e0721b8cf919 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:00.384593 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ef74ac-db3c-4a9b-9811-e0721b8cf919 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:00.384788 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ef74ac-db3c-4a9b-9811-e0721b8cf919 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:00.386720 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 704/1386] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:38:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:38:00.392890 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47f4648-898e-4dcc-99e0-c3f4912a5202 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:00.393376 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47f4648-898e-4dcc-99e0-c3f4912a5202 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:00.393458 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47f4648-898e-4dcc-99e0-c3f4912a5202 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:00.467627 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-b47f4648-898e-4dcc-99e0-c3f4912a5202 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:38:00.468889 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-b47f4648-898e-4dcc-99e0-c3f4912a5202 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:00.477668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:00.576245 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 683/1387] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:38:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:38:00.868482 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ef74ac-db3c-4a9b-9811-e0721b8cf919 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:38:00.870004 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e7ef74ac-db3c-4a9b-9811-e0721b8cf919 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:00.965035 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:01.047606 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ef74ac-db3c-4a9b-9811-e0721b8cf919 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:38:01.171836 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b65ebea3-0b81-4acb-b585-bd21c47b7c27 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:38:01.172397 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65ebea3-0b81-4acb-b585-bd21c47b7c27 service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:01.172685 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65ebea3-0b81-4acb-b585-bd21c47b7c27 service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:01.172960 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b65ebea3-0b81-4acb-b585-bd21c47b7c27 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:01.174602 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-b65ebea3-0b81-4acb-b585-bd21c47b7c27 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:01.241505 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:01.271769 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b65ebea3-0b81-4acb-b585-bd21c47b7c27 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:38:01.273329 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b65ebea3-0b81-4acb-b585-bd21c47b7c27 service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:38:01.291931 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 705/1388] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:38:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 424 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:38:01.308105 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8135297-d8d4-4e95-b6bc-addc5b6c1ed9 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:01.308429 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8135297-d8d4-4e95-b6bc-addc5b6c1ed9 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:01.308700 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8135297-d8d4-4e95-b6bc-addc5b6c1ed9 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:01.310183 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 684/1389] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:38:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:38:01.316302 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420bfbc3-5904-4973-a3af-6dade03d19f5 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:01.316583 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420bfbc3-5904-4973-a3af-6dade03d19f5 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:01.316856 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420bfbc3-5904-4973-a3af-6dade03d19f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:01.391132 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-420bfbc3-5904-4973-a3af-6dade03d19f5 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:38:01.392405 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-420bfbc3-5904-4973-a3af-6dade03d19f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:01.501479 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 706/1390] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:38:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:38:01.801919 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b244cd90-134b-4d94-a138-687c9616181a None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:01.802423 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b244cd90-134b-4d94-a138-687c9616181a None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:01.802529 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b244cd90-134b-4d94-a138-687c9616181a None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:01.803884 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 685/1391] 10.222.0.25 () {54 vars in 962 bytes} [Fri Oct 28 09:38:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:38:01.809325 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65273434-8845-4c2f-a151-c92d25822e0a None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:01.809644 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65273434-8845-4c2f-a151-c92d25822e0a None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:01.809921 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65273434-8845-4c2f-a151-c92d25822e0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:01.856268 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:01.888259 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-65273434-8845-4c2f-a151-c92d25822e0a None None] MFA Rules not processed for user `f4b2ea9aa31445d69e0c1d96b5657708`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:38:01.889480 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-65273434-8845-4c2f-a151-c92d25822e0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:01.965319 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 707/1392] 10.222.0.25 () {58 vars in 1059 bytes} [Fri Oct 28 09:38:01 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:38:01.974423 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b244cd90-134b-4d94-a138-687c9616181a None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:38:01.976011 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-b244cd90-134b-4d94-a138-687c9616181a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:02.082136 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b244cd90-134b-4d94-a138-687c9616181a None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:38:02.121912 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:02.162913 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47be69dd-39f7-4105-8439-ad9b7fb7bf80 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4b2ea9aa31445d69e0c1d96b5657708', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:38:02.163467 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47be69dd-39f7-4105-8439-ad9b7fb7bf80 service glance] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:02.163746 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47be69dd-39f7-4105-8439-ad9b7fb7bf80 service glance] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:02.164030 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47be69dd-39f7-4105-8439-ad9b7fb7bf80 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:02.165617 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-47be69dd-39f7-4105-8439-ad9b7fb7bf80 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:02.279754 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47be69dd-39f7-4105-8439-ad9b7fb7bf80 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:38:02.281209 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47be69dd-39f7-4105-8439-ad9b7fb7bf80 service glance] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:38:02.299973 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 686/1393] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:38:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:38:03.069465 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ee252b-6d72-4e7a-a074-f2ffda388167 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:03.069971 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ee252b-6d72-4e7a-a074-f2ffda388167 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:03.070082 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ee252b-6d72-4e7a-a074-f2ffda388167 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:03.152256 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-48ee252b-6d72-4e7a-a074-f2ffda388167 None None] MFA Rules not processed for user `97c8877c74194a6bbbf25b3c8f5b7dbd`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:38:03.153479 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-48ee252b-6d72-4e7a-a074-f2ffda388167 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:03.237375 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:03.245731 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 708/1394] 10.222.0.25 () {58 vars in 1009 bytes} [Fri Oct 28 09:38:03 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:38:03.292752 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab03ccf-8ee5-4476-be34-4ec3c344ce58 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:03.293244 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab03ccf-8ee5-4476-be34-4ec3c344ce58 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:03.293347 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab03ccf-8ee5-4476-be34-4ec3c344ce58 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:03.294732 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 687/1395] 10.222.0.25 () {54 vars in 943 bytes} [Fri Oct 28 09:38:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:38:03.300847 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40758d5c-188d-436c-b398-aa05143a7615 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:03.301133 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40758d5c-188d-436c-b398-aa05143a7615 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:03.301410 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40758d5c-188d-436c-b398-aa05143a7615 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:03.376575 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-40758d5c-188d-436c-b398-aa05143a7615 None None] MFA Rules not processed for user `9c6cffe0d56e45c4b8fb54fbe5df6094`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:38:03.377826 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-40758d5c-188d-436c-b398-aa05143a7615 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:03.492378 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 709/1396] 10.222.0.25 () {58 vars in 1040 bytes} [Fri Oct 28 09:38:03 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:38:03.501609 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab03ccf-8ee5-4476-be34-4ec3c344ce58 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:38:03.503146 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-6ab03ccf-8ee5-4476-be34-4ec3c344ce58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:03.521669 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e2a26f-ab50-408d-8a9b-15bb1fc55c27 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:03.521957 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e2a26f-ab50-408d-8a9b-15bb1fc55c27 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:03.522242 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e2a26f-ab50-408d-8a9b-15bb1fc55c27 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:03.523647 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 710/1397] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:38:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 09:38:03.528791 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0fa281-16b0-4ae5-93c1-0036c7208eaf None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:03.529071 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0fa281-16b0-4ae5-93c1-0036c7208eaf None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:03.529344 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0fa281-16b0-4ae5-93c1-0036c7208eaf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:03.580555 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:03.604530 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-ec0fa281-16b0-4ae5-93c1-0036c7208eaf None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:38:03.605740 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ec0fa281-16b0-4ae5-93c1-0036c7208eaf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:03.607760 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:03.647361 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab03ccf-8ee5-4476-be34-4ec3c344ce58 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:38:03.707087 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 711/1398] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:38:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:38:03.749353 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc8a5891-9654-41ee-9a94-dce36f12230a service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c6cffe0d56e45c4b8fb54fbe5df6094', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:38:03.749952 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8a5891-9654-41ee-9a94-dce36f12230a service swift] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:03.750281 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8a5891-9654-41ee-9a94-dce36f12230a service swift] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:03.751134 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8a5891-9654-41ee-9a94-dce36f12230a service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:03.752743 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-fc8a5891-9654-41ee-9a94-dce36f12230a service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:03.804702 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:03.836152 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc8a5891-9654-41ee-9a94-dce36f12230a service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:38:03.837744 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc8a5891-9654-41ee-9a94-dce36f12230a service swift] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:38:03.846235 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 688/1399] 10.222.0.25 () {60 vars in 1384 bytes} [Fri Oct 28 09:38:03 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 345 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 28 09:38:04.001722 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0fa281-16b0-4ae5-93c1-0036c7208eaf None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:38:04.003002 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ec0fa281-16b0-4ae5-93c1-0036c7208eaf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:04.131486 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0fa281-16b0-4ae5-93c1-0036c7208eaf None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:38:04.156310 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:04.259537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1818bd3b-579b-4557-b974-d4032062bb1a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:38:04.260507 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1818bd3b-579b-4557-b974-d4032062bb1a service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:04.260792 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1818bd3b-579b-4557-b974-d4032062bb1a service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:04.261102 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1818bd3b-579b-4557-b974-d4032062bb1a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:04.262748 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-1818bd3b-579b-4557-b974-d4032062bb1a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:04.362582 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1818bd3b-579b-4557-b974-d4032062bb1a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:38:04.364171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1818bd3b-579b-4557-b974-d4032062bb1a service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:38:04.382278 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 712/1400] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:38:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:38:06.706015 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e7c0ab-4bc9-4d6d-8205-5162b2b5727b None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:06.707031 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e7c0ab-4bc9-4d6d-8205-5162b2b5727b None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:06.707031 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e7c0ab-4bc9-4d6d-8205-5162b2b5727b None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:06.709267 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 689/1401] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:38:06 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:38:06.718118 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186bb221-9c93-4848-8bb5-3d5d5ad898bd None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:06.720582 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186bb221-9c93-4848-8bb5-3d5d5ad898bd None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:06.721290 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186bb221-9c93-4848-8bb5-3d5d5ad898bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:06.781484 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:06.827214 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-186bb221-9c93-4848-8bb5-3d5d5ad898bd None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:38:06.829265 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-186bb221-9c93-4848-8bb5-3d5d5ad898bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:07.014452 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 713/1402] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:38:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 287 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:38:07.504548 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e7c0ab-4bc9-4d6d-8205-5162b2b5727b None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:38:07.506295 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-81e7c0ab-4bc9-4d6d-8205-5162b2b5727b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:07.794361 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e7c0ab-4bc9-4d6d-8205-5162b2b5727b None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:38:07.881914 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:07.980496 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb77563-4b58-477f-ac62-1626a9b7b90e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:38:07.981181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb77563-4b58-477f-ac62-1626a9b7b90e service nova] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:07.986266 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb77563-4b58-477f-ac62-1626a9b7b90e service nova] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:07.986266 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb77563-4b58-477f-ac62-1626a9b7b90e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:07.987619 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-0fb77563-4b58-477f-ac62-1626a9b7b90e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:08.126135 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fb77563-4b58-477f-ac62-1626a9b7b90e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:38:08.127792 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fb77563-4b58-477f-ac62-1626a9b7b90e service nova] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:38:08.169779 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 690/1403] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:38:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 666 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:38:11.013708 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc279b76-e0ea-4b18-96a1-bc6f29d071c2 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:11.014284 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc279b76-e0ea-4b18-96a1-bc6f29d071c2 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:11.014284 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc279b76-e0ea-4b18-96a1-bc6f29d071c2 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:11.016338 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 714/1404] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:38:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:38:11.023298 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b3e89a-b42d-40c2-89ef-815a0662aadf None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:11.024098 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b3e89a-b42d-40c2-89ef-815a0662aadf None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:11.024538 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b3e89a-b42d-40c2-89ef-815a0662aadf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:11.150165 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-70b3e89a-b42d-40c2-89ef-815a0662aadf None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:38:11.152230 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-70b3e89a-b42d-40c2-89ef-815a0662aadf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:11.155586 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:11.362446 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 691/1405] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:38:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 338 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:38:11.661253 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc279b76-e0ea-4b18-96a1-bc6f29d071c2 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:38:11.663130 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-dc279b76-e0ea-4b18-96a1-bc6f29d071c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:11.793130 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:11.871511 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc279b76-e0ea-4b18-96a1-bc6f29d071c2 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:38:12.082079 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5adcc7e-d4e0-4cf4-a912-16e43b489550 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:38:12.083367 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5adcc7e-d4e0-4cf4-a912-16e43b489550 service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:12.084103 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5adcc7e-d4e0-4cf4-a912-16e43b489550 service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:12.084577 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5adcc7e-d4e0-4cf4-a912-16e43b489550 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:12.088020 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e5adcc7e-d4e0-4cf4-a912-16e43b489550 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:12.211828 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:12.246875 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5adcc7e-d4e0-4cf4-a912-16e43b489550 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:38:12.248482 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5adcc7e-d4e0-4cf4-a912-16e43b489550 service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:38:12.275736 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 715/1406] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:38:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 615 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:38:15.155851 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9c7860-f05e-401d-97b7-c594e925c137 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:15.157445 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9c7860-f05e-401d-97b7-c594e925c137 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:15.157445 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9c7860-f05e-401d-97b7-c594e925c137 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:15.161902 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 692/1407] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:38:15 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:38:15.165023 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81657fbf-20a5-4bad-9866-fb3477e0ddae None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:15.165937 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81657fbf-20a5-4bad-9866-fb3477e0ddae None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:15.165937 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81657fbf-20a5-4bad-9866-fb3477e0ddae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:15.276122 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-81657fbf-20a5-4bad-9866-fb3477e0ddae None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:38:15.277929 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-81657fbf-20a5-4bad-9866-fb3477e0ddae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:15.444324 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 716/1408] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:38:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 280 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:38:15.827626 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf9c7860-f05e-401d-97b7-c594e925c137 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:38:15.830362 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-cf9c7860-f05e-401d-97b7-c594e925c137 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:16.030232 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:16.059380 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf9c7860-f05e-401d-97b7-c594e925c137 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:38:16.231244 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7066669-da5a-4302-a644-50dec167968a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:38:16.231953 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7066669-da5a-4302-a644-50dec167968a service nova] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:16.232294 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7066669-da5a-4302-a644-50dec167968a service nova] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:16.232636 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7066669-da5a-4302-a644-50dec167968a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:16.235076 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-f7066669-da5a-4302-a644-50dec167968a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:16.360140 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7066669-da5a-4302-a644-50dec167968a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:38:16.361774 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7066669-da5a-4302-a644-50dec167968a service nova] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:38:16.368421 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:16.394161 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 693/1409] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:38:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 568 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:38:25.442016 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=81 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:26.391264 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=4 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:28.978901 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfef939-556a-40e0-927e-dd916c90c542 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:28.979457 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfef939-556a-40e0-927e-dd916c90c542 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:28.979457 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfef939-556a-40e0-927e-dd916c90c542 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:28.980867 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 717/1410] 10.222.0.25 () {54 vars in 918 bytes} [Fri Oct 28 09:38:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:38:28.987683 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca876129-32b7-4390-9cf3-f4a047d98c43 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:28.988164 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca876129-32b7-4390-9cf3-f4a047d98c43 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:28.988287 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca876129-32b7-4390-9cf3-f4a047d98c43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:29.198655 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-ca876129-32b7-4390-9cf3-f4a047d98c43 None None] MFA Rules not processed for user `3f1d95049d2d4593bf52670b1a0035fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:38:29.199910 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-ca876129-32b7-4390-9cf3-f4a047d98c43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:29.343036 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 694/1411] 10.222.0.25 () {58 vars in 1015 bytes} [Fri Oct 28 09:38:28 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 357 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:38:29.358948 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837cbf0a-8d23-4ba6-bf3b-19f28d637faf None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:29.359284 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837cbf0a-8d23-4ba6-bf3b-19f28d637faf None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:29.359518 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837cbf0a-8d23-4ba6-bf3b-19f28d637faf None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:29.361010 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 718/1412] 10.222.0.25 () {54 vars in 918 bytes} [Fri Oct 28 09:38:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:38:29.369878 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144711af-093d-46e2-9e49-c0b5c2e7210b None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:29.370178 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144711af-093d-46e2-9e49-c0b5c2e7210b None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:29.370843 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144711af-093d-46e2-9e49-c0b5c2e7210b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:29.465020 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:29.468907 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-144711af-093d-46e2-9e49-c0b5c2e7210b None None] MFA Rules not processed for user `688ceb201d304c2da5373c3cf36c2d79`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:38:29.470605 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-144711af-093d-46e2-9e49-c0b5c2e7210b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:29.598270 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 695/1413] 10.222.0.25 () {58 vars in 1015 bytes} [Fri Oct 28 09:38:29 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 233 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:38:32.404056 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a53d361-1351-460f-a659-90df922b07bb None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:32.404682 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a53d361-1351-460f-a659-90df922b07bb None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:32.404886 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a53d361-1351-460f-a659-90df922b07bb None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:32.406970 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 719/1414] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 09:38:32 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:38:32.413523 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf07c88-343d-4042-9ab5-dbae3cc1dba2 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:32.414053 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf07c88-343d-4042-9ab5-dbae3cc1dba2 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:32.414198 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bf07c88-343d-4042-9ab5-dbae3cc1dba2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:32.487822 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-9bf07c88-343d-4042-9ab5-dbae3cc1dba2 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:38:32.489052 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-9bf07c88-343d-4042-9ab5-dbae3cc1dba2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:32.548724 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:32.600187 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 696/1415] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 09:38:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:38:32.901121 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a53d361-1351-460f-a659-90df922b07bb None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:38:32.902878 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-3a53d361-1351-460f-a659-90df922b07bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:33.083745 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a53d361-1351-460f-a659-90df922b07bb None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:38:33.177252 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:33.241312 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046636bd-69fc-4727-bb99-48f9f2ece709 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:38:33.242062 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046636bd-69fc-4727-bb99-48f9f2ece709 service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:33.242359 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046636bd-69fc-4727-bb99-48f9f2ece709 service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:33.242675 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046636bd-69fc-4727-bb99-48f9f2ece709 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:33.244534 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-046636bd-69fc-4727-bb99-48f9f2ece709 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:33.347775 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046636bd-69fc-4727-bb99-48f9f2ece709 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:38:33.349454 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-046636bd-69fc-4727-bb99-48f9f2ece709 service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:38:33.368393 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 720/1416] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:38:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 468 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:38:34.740956 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d56472a-b53f-4cbd-bb9d-48fdedf23ea5 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:34.741532 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d56472a-b53f-4cbd-bb9d-48fdedf23ea5 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:34.741532 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d56472a-b53f-4cbd-bb9d-48fdedf23ea5 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:34.743140 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 697/1417] 127.0.0.1 () {54 vars in 901 bytes} [Fri Oct 28 09:38:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:38:34.753725 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3fc55bc-add9-4580-97e3-f5b4280d07ad None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:34.754515 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3fc55bc-add9-4580-97e3-f5b4280d07ad None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:34.754515 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3fc55bc-add9-4580-97e3-f5b4280d07ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:34.858117 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-e3fc55bc-add9-4580-97e3-f5b4280d07ad None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:38:34.860031 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e3fc55bc-add9-4580-97e3-f5b4280d07ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:34.889766 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:34.974247 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 721/1418] 127.0.0.1 () {58 vars in 998 bytes} [Fri Oct 28 09:38:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:38:35.134284 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d56472a-b53f-4cbd-bb9d-48fdedf23ea5 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:38:35.136434 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-5d56472a-b53f-4cbd-bb9d-48fdedf23ea5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:35.316023 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d56472a-b53f-4cbd-bb9d-48fdedf23ea5 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:38:35.364480 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:35.443995 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8718698b-3366-4ddc-b34a-ddd29ecec366 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:38:35.444502 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8718698b-3366-4ddc-b34a-ddd29ecec366 service nova] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:35.444784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8718698b-3366-4ddc-b34a-ddd29ecec366 service nova] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:35.445058 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8718698b-3366-4ddc-b34a-ddd29ecec366 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:35.446757 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-8718698b-3366-4ddc-b34a-ddd29ecec366 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:35.546362 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8718698b-3366-4ddc-b34a-ddd29ecec366 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:38:35.547990 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8718698b-3366-4ddc-b34a-ddd29ecec366 service nova] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:38:35.568187 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 698/1419] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:38:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 435 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:38:36.803799 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa09551-53b0-457e-9334-ef346cd1ffee None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:36.804367 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa09551-53b0-457e-9334-ef346cd1ffee None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:36.804367 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa09551-53b0-457e-9334-ef346cd1ffee None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:36.805705 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 722/1420] 127.0.0.1 () {54 vars in 901 bytes} [Fri Oct 28 09:38:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:38:36.813168 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5b4151-8a3d-47b2-8f45-1d2c1c111ebd None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:36.813683 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5b4151-8a3d-47b2-8f45-1d2c1c111ebd None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:36.813683 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f5b4151-8a3d-47b2-8f45-1d2c1c111ebd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:36.868304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:36.890157 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-7f5b4151-8a3d-47b2-8f45-1d2c1c111ebd None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:38:36.891408 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-7f5b4151-8a3d-47b2-8f45-1d2c1c111ebd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:36.991051 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 699/1421] 127.0.0.1 () {58 vars in 998 bytes} [Fri Oct 28 09:38:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:38:37.144210 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa09551-53b0-457e-9334-ef346cd1ffee None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:38:37.146163 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-baa09551-53b0-457e-9334-ef346cd1ffee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:37.302672 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa09551-53b0-457e-9334-ef346cd1ffee None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:38:37.315373 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:37.415796 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395af7a2-0d5d-4034-844c-5148abdebdfa service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:38:37.416322 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395af7a2-0d5d-4034-844c-5148abdebdfa service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:37.416612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395af7a2-0d5d-4034-844c-5148abdebdfa service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:37.416892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395af7a2-0d5d-4034-844c-5148abdebdfa service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:37.418513 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-395af7a2-0d5d-4034-844c-5148abdebdfa service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:37.512728 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-395af7a2-0d5d-4034-844c-5148abdebdfa service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:38:37.514317 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-395af7a2-0d5d-4034-844c-5148abdebdfa service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:38:37.533529 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 723/1422] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:38:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 391 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:38:38.677697 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a3a4cb-127f-40f6-b2cb-3597a69fa532 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:38.678263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a3a4cb-127f-40f6-b2cb-3597a69fa532 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:38.678263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a3a4cb-127f-40f6-b2cb-3597a69fa532 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:38.679636 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 700/1423] 127.0.0.1 () {54 vars in 901 bytes} [Fri Oct 28 09:38:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:38:38.686219 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641de985-dd95-47d4-a7ad-351c49609b43 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:38.687289 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641de985-dd95-47d4-a7ad-351c49609b43 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:38.687408 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641de985-dd95-47d4-a7ad-351c49609b43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:38.707723 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:38.766516 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-641de985-dd95-47d4-a7ad-351c49609b43 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:38:38.767750 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-641de985-dd95-47d4-a7ad-351c49609b43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:38.865782 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 724/1424] 127.0.0.1 () {58 vars in 998 bytes} [Fri Oct 28 09:38:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:38:39.016009 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a3a4cb-127f-40f6-b2cb-3597a69fa532 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:38:39.017615 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-62a3a4cb-127f-40f6-b2cb-3597a69fa532 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:39.131848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:39.176694 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a3a4cb-127f-40f6-b2cb-3597a69fa532 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:38:39.295946 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5715fea4-8f19-4607-ace6-44f9399ea977 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:38:39.297157 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5715fea4-8f19-4607-ace6-44f9399ea977 service nova] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:39.297433 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5715fea4-8f19-4607-ace6-44f9399ea977 service nova] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:39.297703 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5715fea4-8f19-4607-ace6-44f9399ea977 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:39.299340 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-5715fea4-8f19-4607-ace6-44f9399ea977 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:39.376351 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:39.392052 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5715fea4-8f19-4607-ace6-44f9399ea977 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:38:39.393568 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5715fea4-8f19-4607-ace6-44f9399ea977 service nova] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:38:39.411872 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 701/1425] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:38:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 396 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:38:40.540348 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc0ec50-babb-45e1-aafb-0c2034542d6f None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:40.540915 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc0ec50-babb-45e1-aafb-0c2034542d6f None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:40.540915 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbc0ec50-babb-45e1-aafb-0c2034542d6f None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:40.542197 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 725/1426] 127.0.0.1 () {54 vars in 901 bytes} [Fri Oct 28 09:38:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:38:40.549901 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345f1268-2828-40f0-9542-bc1171681f22 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:40.550501 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345f1268-2828-40f0-9542-bc1171681f22 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:40.550501 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345f1268-2828-40f0-9542-bc1171681f22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:40.631988 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-345f1268-2828-40f0-9542-bc1171681f22 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:38:40.633194 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-345f1268-2828-40f0-9542-bc1171681f22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:40.727797 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 702/1427] 127.0.0.1 () {58 vars in 998 bytes} [Fri Oct 28 09:38:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:38:40.880487 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc0ec50-babb-45e1-aafb-0c2034542d6f None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:38:40.881986 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-dbc0ec50-babb-45e1-aafb-0c2034542d6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:40.966856 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:41.029786 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbc0ec50-babb-45e1-aafb-0c2034542d6f None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:38:41.147423 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c3c5f2-e2e7-448d-bc88-28913e6e750f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:38:41.148030 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c3c5f2-e2e7-448d-bc88-28913e6e750f service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:41.148304 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c3c5f2-e2e7-448d-bc88-28913e6e750f service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:41.148581 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c3c5f2-e2e7-448d-bc88-28913e6e750f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:41.150303 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-46c3c5f2-e2e7-448d-bc88-28913e6e750f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:41.199718 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:41.242724 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c3c5f2-e2e7-448d-bc88-28913e6e750f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:38:41.244297 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c3c5f2-e2e7-448d-bc88-28913e6e750f service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:38:41.262205 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 726/1428] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:38:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:38:42.320413 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dbcf7f1-d9c9-4ba3-acf0-ed9bbca4ec93 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:42.320985 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dbcf7f1-d9c9-4ba3-acf0-ed9bbca4ec93 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:42.320985 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dbcf7f1-d9c9-4ba3-acf0-ed9bbca4ec93 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:42.322296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 703/1429] 127.0.0.1 () {54 vars in 901 bytes} [Fri Oct 28 09:38:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:38:42.329267 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30436ad6-0d9d-466e-8a44-88d053de29a0 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:42.329766 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30436ad6-0d9d-466e-8a44-88d053de29a0 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:42.329877 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30436ad6-0d9d-466e-8a44-88d053de29a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:42.406614 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-30436ad6-0d9d-466e-8a44-88d053de29a0 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:38:42.407739 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-30436ad6-0d9d-466e-8a44-88d053de29a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:42.509523 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 727/1430] 127.0.0.1 () {58 vars in 998 bytes} [Fri Oct 28 09:38:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:38:42.659779 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dbcf7f1-d9c9-4ba3-acf0-ed9bbca4ec93 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:38:42.661216 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-4dbcf7f1-d9c9-4ba3-acf0-ed9bbca4ec93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:42.709562 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:42.804664 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dbcf7f1-d9c9-4ba3-acf0-ed9bbca4ec93 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:38:42.916560 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe12918f-9b6d-45aa-b3a5-b53bc1a9a545 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:38:42.917079 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe12918f-9b6d-45aa-b3a5-b53bc1a9a545 service nova] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:42.917359 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe12918f-9b6d-45aa-b3a5-b53bc1a9a545 service nova] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:42.917632 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe12918f-9b6d-45aa-b3a5-b53bc1a9a545 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:42.919268 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-fe12918f-9b6d-45aa-b3a5-b53bc1a9a545 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:42.926981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:43.010531 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe12918f-9b6d-45aa-b3a5-b53bc1a9a545 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:38:43.012131 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe12918f-9b6d-45aa-b3a5-b53bc1a9a545 service nova] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:38:43.031493 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 704/1431] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:38:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:38:44.110615 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4aefe7d-5a8b-4652-9f7a-c0a8789463d9 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:44.111170 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4aefe7d-5a8b-4652-9f7a-c0a8789463d9 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:44.111170 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4aefe7d-5a8b-4652-9f7a-c0a8789463d9 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:44.112535 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 728/1432] 127.0.0.1 () {54 vars in 901 bytes} [Fri Oct 28 09:38:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:38:44.119277 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780cfd3a-0806-4ad1-8f3a-02115adc10fa None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:44.120448 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780cfd3a-0806-4ad1-8f3a-02115adc10fa None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:44.120448 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780cfd3a-0806-4ad1-8f3a-02115adc10fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:44.199162 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-780cfd3a-0806-4ad1-8f3a-02115adc10fa None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:38:44.200415 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-780cfd3a-0806-4ad1-8f3a-02115adc10fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:44.272403 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:44.302119 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 705/1433] 127.0.0.1 () {58 vars in 998 bytes} [Fri Oct 28 09:38:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:38:44.453416 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4aefe7d-5a8b-4652-9f7a-c0a8789463d9 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:38:44.454871 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a4aefe7d-5a8b-4652-9f7a-c0a8789463d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:44.473143 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:44.604408 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4aefe7d-5a8b-4652-9f7a-c0a8789463d9 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:38:44.692231 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:44.720039 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ab1247-07cf-4093-adf6-e8f38cae0e89 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:38:44.720562 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ab1247-07cf-4093-adf6-e8f38cae0e89 service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:44.720839 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ab1247-07cf-4093-adf6-e8f38cae0e89 service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:44.721110 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ab1247-07cf-4093-adf6-e8f38cae0e89 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:44.722719 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-91ab1247-07cf-4093-adf6-e8f38cae0e89 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:44.814666 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91ab1247-07cf-4093-adf6-e8f38cae0e89 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:38:44.816293 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91ab1247-07cf-4093-adf6-e8f38cae0e89 service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:38:44.834837 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 729/1434] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:38:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:38:45.898565 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6925da-3c78-437a-8c91-b969bb0eafee None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:45.899155 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6925da-3c78-437a-8c91-b969bb0eafee None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:45.899155 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb6925da-3c78-437a-8c91-b969bb0eafee None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:45.900543 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 706/1435] 127.0.0.1 () {54 vars in 901 bytes} [Fri Oct 28 09:38:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:38:45.907765 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bccbc181-a519-4c93-92d9-46d8e7995981 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:45.908325 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bccbc181-a519-4c93-92d9-46d8e7995981 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:45.908325 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bccbc181-a519-4c93-92d9-46d8e7995981 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:45.982961 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-bccbc181-a519-4c93-92d9-46d8e7995981 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:38:45.984194 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-bccbc181-a519-4c93-92d9-46d8e7995981 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:46.025301 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:46.081204 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 730/1436] 127.0.0.1 () {58 vars in 998 bytes} [Fri Oct 28 09:38:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:38:46.229632 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6925da-3c78-437a-8c91-b969bb0eafee None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:38:46.232278 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-eb6925da-3c78-437a-8c91-b969bb0eafee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:46.376551 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb6925da-3c78-437a-8c91-b969bb0eafee None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:38:46.430252 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:46.489957 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fca7c18a-fa56-43ca-8cf5-08373eaf6774 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:38:46.490514 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca7c18a-fa56-43ca-8cf5-08373eaf6774 service nova] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:46.490796 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca7c18a-fa56-43ca-8cf5-08373eaf6774 service nova] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:46.491070 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca7c18a-fa56-43ca-8cf5-08373eaf6774 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:46.492657 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-fca7c18a-fa56-43ca-8cf5-08373eaf6774 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:46.583541 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fca7c18a-fa56-43ca-8cf5-08373eaf6774 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:38:46.585125 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fca7c18a-fa56-43ca-8cf5-08373eaf6774 service nova] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:38:46.603075 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 707/1437] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:38:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:38:47.670290 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b170ad-4ca5-4550-bf51-97cf40065d10 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:47.670856 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b170ad-4ca5-4550-bf51-97cf40065d10 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:47.670978 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b170ad-4ca5-4550-bf51-97cf40065d10 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:47.672291 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 731/1438] 127.0.0.1 () {54 vars in 901 bytes} [Fri Oct 28 09:38:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 09:38:47.679182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37c4450-b0be-4bb0-9a63-a55925a9774c None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:47.679715 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37c4450-b0be-4bb0-9a63-a55925a9774c None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:47.679715 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37c4450-b0be-4bb0-9a63-a55925a9774c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:47.756087 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-c37c4450-b0be-4bb0-9a63-a55925a9774c None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:38:47.757292 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-c37c4450-b0be-4bb0-9a63-a55925a9774c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:47.759342 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:47.855515 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 708/1439] 127.0.0.1 () {58 vars in 998 bytes} [Fri Oct 28 09:38:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:38:48.007706 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b170ad-4ca5-4550-bf51-97cf40065d10 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:38:48.009266 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-c6b170ad-4ca5-4550-bf51-97cf40065d10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:48.183660 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b170ad-4ca5-4550-bf51-97cf40065d10 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:38:48.218738 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:48.310165 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07265d52-5ac9-46d5-918f-72fb11f3c8d9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:38:48.310801 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07265d52-5ac9-46d5-918f-72fb11f3c8d9 service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:48.311085 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07265d52-5ac9-46d5-918f-72fb11f3c8d9 service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:48.311361 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07265d52-5ac9-46d5-918f-72fb11f3c8d9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:48.313488 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-07265d52-5ac9-46d5-918f-72fb11f3c8d9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:48.404277 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07265d52-5ac9-46d5-918f-72fb11f3c8d9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:38:48.405917 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07265d52-5ac9-46d5-918f-72fb11f3c8d9 service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:38:48.424102 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 732/1440] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:38:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 417 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:38:50.493972 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0707b9-0e00-4ba9-b324-de046f5bb046 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:50.494551 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0707b9-0e00-4ba9-b324-de046f5bb046 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:50.494551 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0707b9-0e00-4ba9-b324-de046f5bb046 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:50.495817 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 709/1441] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 09:38:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:38:50.501677 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978a193f-b4bb-4a61-b179-d294eb61e8c8 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:50.502096 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978a193f-b4bb-4a61-b179-d294eb61e8c8 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:50.502215 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978a193f-b4bb-4a61-b179-d294eb61e8c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:50.544366 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:50.580256 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-978a193f-b4bb-4a61-b179-d294eb61e8c8 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:38:50.582237 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-978a193f-b4bb-4a61-b179-d294eb61e8c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:50.705743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 733/1442] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 09:38:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:38:50.993141 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e0707b9-0e00-4ba9-b324-de046f5bb046 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:38:50.994695 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-2e0707b9-0e00-4ba9-b324-de046f5bb046 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:51.099311 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e0707b9-0e00-4ba9-b324-de046f5bb046 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:38:51.132443 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:51.176071 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae76d4b-3886-4bfc-ae79-e7b310425596 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:38:51.176674 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae76d4b-3886-4bfc-ae79-e7b310425596 service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:51.176950 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae76d4b-3886-4bfc-ae79-e7b310425596 service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:51.177219 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae76d4b-3886-4bfc-ae79-e7b310425596 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:51.178837 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-aae76d4b-3886-4bfc-ae79-e7b310425596 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:51.288426 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae76d4b-3886-4bfc-ae79-e7b310425596 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:38:51.289779 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae76d4b-3886-4bfc-ae79-e7b310425596 service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:38:51.308170 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 710/1443] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:38:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:38:53.629064 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998aa777-a45d-4a58-961b-8ff97909d465 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:53.629600 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998aa777-a45d-4a58-961b-8ff97909d465 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:53.629600 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-998aa777-a45d-4a58-961b-8ff97909d465 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:53.631027 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 734/1444] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 09:38:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:38:53.636349 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa68fa4-a77f-4712-aebc-8e1a6cb4150b None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:53.636776 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa68fa4-a77f-4712-aebc-8e1a6cb4150b None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:53.636892 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa68fa4-a77f-4712-aebc-8e1a6cb4150b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:53.704347 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-8fa68fa4-a77f-4712-aebc-8e1a6cb4150b None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:38:53.705568 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-8fa68fa4-a77f-4712-aebc-8e1a6cb4150b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:53.718937 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:53.805594 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 711/1445] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 09:38:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:38:54.089840 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-998aa777-a45d-4a58-961b-8ff97909d465 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:38:54.091395 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-998aa777-a45d-4a58-961b-8ff97909d465 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:54.196865 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-998aa777-a45d-4a58-961b-8ff97909d465 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:38:54.209713 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:54.271465 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f550685-0b09-489a-ab73-d5b67a4a2dbf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:38:54.272066 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f550685-0b09-489a-ab73-d5b67a4a2dbf service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:54.272345 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f550685-0b09-489a-ab73-d5b67a4a2dbf service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:54.272617 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f550685-0b09-489a-ab73-d5b67a4a2dbf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:54.274199 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-9f550685-0b09-489a-ab73-d5b67a4a2dbf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:54.385827 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f550685-0b09-489a-ab73-d5b67a4a2dbf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:38:54.387181 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f550685-0b09-489a-ab73-d5b67a4a2dbf service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:38:54.405501 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 735/1446] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:38:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:38:56.898796 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940e6d90-4634-4812-baa0-517700280a94 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:56.899392 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940e6d90-4634-4812-baa0-517700280a94 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:56.899392 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940e6d90-4634-4812-baa0-517700280a94 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:56.900952 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 712/1447] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 09:38:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:38:56.907724 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff21127-d08b-4b95-af4a-8766f6499120 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:56.908250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff21127-d08b-4b95-af4a-8766f6499120 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:56.908250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff21127-d08b-4b95-af4a-8766f6499120 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:56.961118 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:56.981730 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-8ff21127-d08b-4b95-af4a-8766f6499120 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:38:56.983855 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-8ff21127-d08b-4b95-af4a-8766f6499120 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:57.087219 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 736/1448] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 09:38:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:38:57.374756 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940e6d90-4634-4812-baa0-517700280a94 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:38:57.376286 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-940e6d90-4634-4812-baa0-517700280a94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:57.482742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940e6d90-4634-4812-baa0-517700280a94 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:38:57.523845 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:38:57.559354 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46996ba4-f1a4-4a00-bd89-f133a6cfc5b8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:38:57.559967 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46996ba4-f1a4-4a00-bd89-f133a6cfc5b8 service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:38:57.560671 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46996ba4-f1a4-4a00-bd89-f133a6cfc5b8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:38:57.560671 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46996ba4-f1a4-4a00-bd89-f133a6cfc5b8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:38:57.563072 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-46996ba4-f1a4-4a00-bd89-f133a6cfc5b8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:38:57.678917 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46996ba4-f1a4-4a00-bd89-f133a6cfc5b8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:38:57.680270 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46996ba4-f1a4-4a00-bd89-f133a6cfc5b8 service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:38:57.698365 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 713/1449] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:38:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:39:00.051454 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f159696-32ca-4c12-a673-cf2973ea7127 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:39:00.052035 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f159696-32ca-4c12-a673-cf2973ea7127 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:39:00.052035 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f159696-32ca-4c12-a673-cf2973ea7127 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:39:00.053376 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 737/1450] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 09:39:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:39:00.059292 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9cdc9bc-74a8-4a36-a1aa-391c0ba5e5e2 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:39:00.059768 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9cdc9bc-74a8-4a36-a1aa-391c0ba5e5e2 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:39:00.059867 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9cdc9bc-74a8-4a36-a1aa-391c0ba5e5e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:39:00.132873 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-f9cdc9bc-74a8-4a36-a1aa-391c0ba5e5e2 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:39:00.134110 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-f9cdc9bc-74a8-4a36-a1aa-391c0ba5e5e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:39:00.156048 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:39:00.236685 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 714/1451] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 09:39:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:39:00.528396 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f159696-32ca-4c12-a673-cf2973ea7127 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:39:00.529981 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-3f159696-32ca-4c12-a673-cf2973ea7127 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:39:00.640675 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f159696-32ca-4c12-a673-cf2973ea7127 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:39:00.663549 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:39:00.726579 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-127f6cce-d0b4-453f-8562-05e5724851f9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:39:00.727216 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127f6cce-d0b4-453f-8562-05e5724851f9 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:39:00.727487 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127f6cce-d0b4-453f-8562-05e5724851f9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:39:00.727870 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127f6cce-d0b4-453f-8562-05e5724851f9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:39:00.729406 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-127f6cce-d0b4-453f-8562-05e5724851f9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:39:00.851018 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-127f6cce-d0b4-453f-8562-05e5724851f9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:39:00.852663 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-127f6cce-d0b4-453f-8562-05e5724851f9 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:39:00.872651 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 738/1452] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:39:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 345 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:39:03.213207 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610efd29-6e35-442d-a7a2-7031717b6f44 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:39:03.213881 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610efd29-6e35-442d-a7a2-7031717b6f44 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:39:03.213881 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610efd29-6e35-442d-a7a2-7031717b6f44 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:39:03.215224 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 715/1453] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 09:39:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:39:03.221354 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b692a8f1-a6e3-4a71-9e74-d0759ce481e9 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:39:03.221887 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b692a8f1-a6e3-4a71-9e74-d0759ce481e9 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:39:03.221887 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b692a8f1-a6e3-4a71-9e74-d0759ce481e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:39:03.296210 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:39:03.301398 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-b692a8f1-a6e3-4a71-9e74-d0759ce481e9 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:39:03.303578 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-b692a8f1-a6e3-4a71-9e74-d0759ce481e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:39:03.432141 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 739/1454] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 09:39:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:39:03.757854 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610efd29-6e35-442d-a7a2-7031717b6f44 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:39:03.759660 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-610efd29-6e35-442d-a7a2-7031717b6f44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:39:03.877773 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610efd29-6e35-442d-a7a2-7031717b6f44 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:39:03.937548 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:39:03.971066 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-464d1c99-a015-441f-9046-247b2cdc2051 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:39:03.971691 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464d1c99-a015-441f-9046-247b2cdc2051 service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:39:03.971969 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464d1c99-a015-441f-9046-247b2cdc2051 service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:39:03.972242 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-464d1c99-a015-441f-9046-247b2cdc2051 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:39:03.973882 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-464d1c99-a015-441f-9046-247b2cdc2051 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:39:04.105464 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-464d1c99-a015-441f-9046-247b2cdc2051 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:39:04.106955 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-464d1c99-a015-441f-9046-247b2cdc2051 service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:39:04.127738 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 716/1455] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:39:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:39:06.675418 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b64162a-3a5e-4c2e-8bc3-0d731311d62b None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:39:06.676047 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b64162a-3a5e-4c2e-8bc3-0d731311d62b None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:39:06.676047 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b64162a-3a5e-4c2e-8bc3-0d731311d62b None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:39:06.677509 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 740/1456] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 09:39:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:39:06.692009 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c7505f-7d67-436c-8c98-9ab927f1a074 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:39:06.692618 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c7505f-7d67-436c-8c98-9ab927f1a074 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:39:06.692618 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c7505f-7d67-436c-8c98-9ab927f1a074 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:39:06.779334 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-51c7505f-7d67-436c-8c98-9ab927f1a074 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:39:06.780615 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-51c7505f-7d67-436c-8c98-9ab927f1a074 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:39:06.814488 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:39:06.907543 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 717/1457] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 09:39:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 217 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:39:07.236951 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b64162a-3a5e-4c2e-8bc3-0d731311d62b None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:39:07.240444 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-0b64162a-3a5e-4c2e-8bc3-0d731311d62b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:39:07.368989 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b64162a-3a5e-4c2e-8bc3-0d731311d62b None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:39:07.402216 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:39:07.465802 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c8cd63-b820-49e2-94e3-966eb80cd96c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:39:07.466800 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c8cd63-b820-49e2-94e3-966eb80cd96c service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:39:07.467755 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c8cd63-b820-49e2-94e3-966eb80cd96c service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:39:07.468198 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c8cd63-b820-49e2-94e3-966eb80cd96c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:39:07.471008 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-82c8cd63-b820-49e2-94e3-966eb80cd96c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:39:07.612647 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c8cd63-b820-49e2-94e3-966eb80cd96c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:39:07.614130 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c8cd63-b820-49e2-94e3-966eb80cd96c service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:39:07.633821 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 741/1458] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:39:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 398 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:39:10.226895 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f4cac0-7221-423c-8b8b-fbf22abf0630 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:39:10.227531 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f4cac0-7221-423c-8b8b-fbf22abf0630 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:39:10.227531 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f4cac0-7221-423c-8b8b-fbf22abf0630 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:39:10.228790 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 718/1459] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 09:39:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:39:10.234705 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a27e5f2-136b-41d0-b75f-e9e10a3f6be6 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:39:10.235211 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a27e5f2-136b-41d0-b75f-e9e10a3f6be6 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:39:10.235309 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a27e5f2-136b-41d0-b75f-e9e10a3f6be6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:39:10.334062 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-3a27e5f2-136b-41d0-b75f-e9e10a3f6be6 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:39:10.335324 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-3a27e5f2-136b-41d0-b75f-e9e10a3f6be6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:39:10.345500 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:39:10.449230 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 742/1460] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 09:39:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:39:10.744989 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f4cac0-7221-423c-8b8b-fbf22abf0630 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:39:10.746666 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-37f4cac0-7221-423c-8b8b-fbf22abf0630 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:39:10.868395 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f4cac0-7221-423c-8b8b-fbf22abf0630 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:39:10.934534 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:39:10.955807 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e2f039-c4b8-45cd-b5ab-03a122fbe365 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:39:10.956426 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e2f039-c4b8-45cd-b5ab-03a122fbe365 service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:39:10.956712 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e2f039-c4b8-45cd-b5ab-03a122fbe365 service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:39:10.956982 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e2f039-c4b8-45cd-b5ab-03a122fbe365 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:39:10.958686 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-70e2f039-c4b8-45cd-b5ab-03a122fbe365 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:39:11.078330 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e2f039-c4b8-45cd-b5ab-03a122fbe365 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:39:11.079804 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e2f039-c4b8-45cd-b5ab-03a122fbe365 service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:39:11.098110 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 719/1461] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:39:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 354 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:39:13.334985 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ed4576-ce9c-4063-9904-be8a193df613 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:39:13.335564 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ed4576-ce9c-4063-9904-be8a193df613 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:39:13.335564 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ed4576-ce9c-4063-9904-be8a193df613 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:39:13.336922 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 743/1462] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 09:39:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:39:13.342677 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83cbde2d-5c8e-4e1e-b019-0c3dd8e17917 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:39:13.343095 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83cbde2d-5c8e-4e1e-b019-0c3dd8e17917 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:39:13.343224 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83cbde2d-5c8e-4e1e-b019-0c3dd8e17917 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:39:13.407701 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-83cbde2d-5c8e-4e1e-b019-0c3dd8e17917 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:39:13.408911 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-83cbde2d-5c8e-4e1e-b019-0c3dd8e17917 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:39:13.446623 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:39:13.503888 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 720/1463] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 09:39:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:39:13.784486 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ed4576-ce9c-4063-9904-be8a193df613 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:39:13.786035 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e8ed4576-ce9c-4063-9904-be8a193df613 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:39:13.888961 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ed4576-ce9c-4063-9904-be8a193df613 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:39:13.926698 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:39:13.962860 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69f89acd-4d8d-4452-b573-7e78b8b7f87f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:39:13.963470 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f89acd-4d8d-4452-b573-7e78b8b7f87f service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:39:13.963751 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f89acd-4d8d-4452-b573-7e78b8b7f87f service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:39:13.964024 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f89acd-4d8d-4452-b573-7e78b8b7f87f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:39:13.965624 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-69f89acd-4d8d-4452-b573-7e78b8b7f87f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:39:14.075090 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69f89acd-4d8d-4452-b573-7e78b8b7f87f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:39:14.076492 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69f89acd-4d8d-4452-b573-7e78b8b7f87f service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:39:14.094743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 744/1464] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:39:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:39:16.307727 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf7604d-696d-40f4-8bf0-f8b0c0fb1ad5 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:39:16.308292 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf7604d-696d-40f4-8bf0-f8b0c0fb1ad5 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:39:16.308292 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf7604d-696d-40f4-8bf0-f8b0c0fb1ad5 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:39:16.309590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 721/1465] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 09:39:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:39:16.314946 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf00870-e825-4891-a316-70f284b0ea78 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:39:16.315393 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf00870-e825-4891-a316-70f284b0ea78 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:39:16.315504 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf00870-e825-4891-a316-70f284b0ea78 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:39:16.389623 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-bcf00870-e825-4891-a316-70f284b0ea78 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:39:16.390931 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-bcf00870-e825-4891-a316-70f284b0ea78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:39:16.413391 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:39:16.498510 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 745/1466] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 09:39:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:39:16.782352 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf7604d-696d-40f4-8bf0-f8b0c0fb1ad5 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:39:16.783917 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-5cf7604d-696d-40f4-8bf0-f8b0c0fb1ad5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:39:16.887055 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf7604d-696d-40f4-8bf0-f8b0c0fb1ad5 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:39:16.953712 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:39:16.961141 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7d9728-50d8-4b55-84cc-d53b964010d0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:39:16.961748 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7d9728-50d8-4b55-84cc-d53b964010d0 service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:39:16.962035 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7d9728-50d8-4b55-84cc-d53b964010d0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:39:16.962305 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7d9728-50d8-4b55-84cc-d53b964010d0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:39:16.963913 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-3f7d9728-50d8-4b55-84cc-d53b964010d0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:39:17.075092 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f7d9728-50d8-4b55-84cc-d53b964010d0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:39:17.076472 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f7d9728-50d8-4b55-84cc-d53b964010d0 service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:39:17.094704 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 722/1467] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:39:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:39:19.359959 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc01adf2-fdad-4f17-9d27-84bd043ee8f4 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:39:19.360547 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc01adf2-fdad-4f17-9d27-84bd043ee8f4 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:39:19.360547 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc01adf2-fdad-4f17-9d27-84bd043ee8f4 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:39:19.361877 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 746/1468] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 09:39:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:39:19.367852 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6963c5e1-adef-4a7b-9a95-05cf27085fe1 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:39:19.368375 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6963c5e1-adef-4a7b-9a95-05cf27085fe1 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:39:19.368494 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6963c5e1-adef-4a7b-9a95-05cf27085fe1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:39:19.442301 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-6963c5e1-adef-4a7b-9a95-05cf27085fe1 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:39:19.443575 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-6963c5e1-adef-4a7b-9a95-05cf27085fe1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:39:19.492634 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:39:19.544683 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 723/1469] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 09:39:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:39:19.845233 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc01adf2-fdad-4f17-9d27-84bd043ee8f4 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:39:19.847887 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-cc01adf2-fdad-4f17-9d27-84bd043ee8f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:39:19.993453 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc01adf2-fdad-4f17-9d27-84bd043ee8f4 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:39:20.062560 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:39:20.101601 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd282e60-bfe6-44b1-abec-03e908ee735d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:39:20.102228 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd282e60-bfe6-44b1-abec-03e908ee735d service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:39:20.102729 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd282e60-bfe6-44b1-abec-03e908ee735d service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:39:20.103079 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd282e60-bfe6-44b1-abec-03e908ee735d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:39:20.104984 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-bd282e60-bfe6-44b1-abec-03e908ee735d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:39:20.234176 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd282e60-bfe6-44b1-abec-03e908ee735d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:39:20.235633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd282e60-bfe6-44b1-abec-03e908ee735d service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:39:20.254296 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 747/1470] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:39:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 410 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:39:29.543094 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=22 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:39:30.251834 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=66 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:39:34.609486 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa3e074-9bde-40f2-9f8f-721a3cd9730d None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:39:34.610101 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa3e074-9bde-40f2-9f8f-721a3cd9730d None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:39:34.610101 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aa3e074-9bde-40f2-9f8f-721a3cd9730d None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:39:34.611433 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 724/1471] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:39:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:39:34.617223 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc366366-699e-4cb9-bb74-0e0287932bff None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:39:34.617742 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc366366-699e-4cb9-bb74-0e0287932bff None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:39:34.617865 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc366366-699e-4cb9-bb74-0e0287932bff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:39:34.688919 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-dc366366-699e-4cb9-bb74-0e0287932bff None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:39:34.690145 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-dc366366-699e-4cb9-bb74-0e0287932bff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:39:34.787504 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 748/1472] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:39:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:39:35.086946 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c94ea62-fabf-4ece-8720-0e3a35b600be None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:39:35.087237 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c94ea62-fabf-4ece-8720-0e3a35b600be None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:39:35.087520 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c94ea62-fabf-4ece-8720-0e3a35b600be None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:39:35.088906 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 725/1473] 10.222.0.25 () {54 vars in 962 bytes} [Fri Oct 28 09:39:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:39:35.094673 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c3e81b-0e3c-40c4-a467-4475f3c71a1e None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:39:35.094960 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c3e81b-0e3c-40c4-a467-4475f3c71a1e None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:39:35.095235 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36c3e81b-0e3c-40c4-a467-4475f3c71a1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:39:35.173599 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-36c3e81b-0e3c-40c4-a467-4475f3c71a1e None None] MFA Rules not processed for user `f4b2ea9aa31445d69e0c1d96b5657708`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:39:35.174854 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-36c3e81b-0e3c-40c4-a467-4475f3c71a1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:39:35.200086 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:39:35.257893 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 749/1474] 10.222.0.25 () {58 vars in 1059 bytes} [Fri Oct 28 09:39:35 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:39:35.266341 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c94ea62-fabf-4ece-8720-0e3a35b600be None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:39:35.267944 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-2c94ea62-fabf-4ece-8720-0e3a35b600be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:39:35.371286 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c94ea62-fabf-4ece-8720-0e3a35b600be None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:39:35.444937 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8544d87b-eeb7-4f6e-a91f-e3937e0a4afa service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4b2ea9aa31445d69e0c1d96b5657708', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:39:35.445546 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8544d87b-eeb7-4f6e-a91f-e3937e0a4afa service glance] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:39:35.445831 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8544d87b-eeb7-4f6e-a91f-e3937e0a4afa service glance] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:39:35.446111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8544d87b-eeb7-4f6e-a91f-e3937e0a4afa service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:39:35.447741 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-8544d87b-eeb7-4f6e-a91f-e3937e0a4afa service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:39:35.501479 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:39:35.563522 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8544d87b-eeb7-4f6e-a91f-e3937e0a4afa service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:39:35.564976 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8544d87b-eeb7-4f6e-a91f-e3937e0a4afa service glance] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:39:35.584803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 726/1475] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:39:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:39:37.824698 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b782d2c-cdc7-41e6-af6f-7c8150f8a997 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:39:37.825226 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b782d2c-cdc7-41e6-af6f-7c8150f8a997 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:39:37.825226 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b782d2c-cdc7-41e6-af6f-7c8150f8a997 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:39:37.827187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 750/1476] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:39:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:39:37.832815 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c66b84-e2b7-4038-9bd6-111ae97ecd49 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:39:37.833232 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c66b84-e2b7-4038-9bd6-111ae97ecd49 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:39:37.833408 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c66b84-e2b7-4038-9bd6-111ae97ecd49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:39:37.901885 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-74c66b84-e2b7-4038-9bd6-111ae97ecd49 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:39:37.903140 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-74c66b84-e2b7-4038-9bd6-111ae97ecd49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:39:38.002001 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 727/1477] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:39:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:39:38.273891 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b782d2c-cdc7-41e6-af6f-7c8150f8a997 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:39:38.275477 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-3b782d2c-cdc7-41e6-af6f-7c8150f8a997 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:39:38.408923 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b782d2c-cdc7-41e6-af6f-7c8150f8a997 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:39:38.460940 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:39:38.507591 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f691f95d-a3f2-448b-a8e0-61f851838a1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:39:38.508205 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f691f95d-a3f2-448b-a8e0-61f851838a1d admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:39:38.508512 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f691f95d-a3f2-448b-a8e0-61f851838a1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:39:38.508783 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f691f95d-a3f2-448b-a8e0-61f851838a1d admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:39:38.509424 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f691f95d-a3f2-448b-a8e0-61f851838a1d admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:39:38.511212 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: WARNING py.warnings [None req-f691f95d-a3f2-448b-a8e0-61f851838a1d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 28 09:39:38.511212 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: warnings.warn(msg) Oct 28 09:39:38.511212 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: Oct 28 09:39:38.512352 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f691f95d-a3f2-448b-a8e0-61f851838a1d admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:39:38.537184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 751/1478] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Oct 28 09:39:38 2022] GET /identity/v3/projects => generated 2553 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:39:40.729095 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f617278e-423e-4626-b3f3-29c9949ad4ea None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:39:40.729656 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f617278e-423e-4626-b3f3-29c9949ad4ea None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:39:40.729656 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f617278e-423e-4626-b3f3-29c9949ad4ea None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:39:40.731057 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 728/1479] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:39:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:39:40.737315 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907a76c3-b481-4e7d-a47f-fc673ed98e20 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:39:40.737783 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907a76c3-b481-4e7d-a47f-fc673ed98e20 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:39:40.737783 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907a76c3-b481-4e7d-a47f-fc673ed98e20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:39:40.807818 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-907a76c3-b481-4e7d-a47f-fc673ed98e20 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:39:40.809395 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-907a76c3-b481-4e7d-a47f-fc673ed98e20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:39:40.911293 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 752/1480] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:39:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:39:41.203265 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f617278e-423e-4626-b3f3-29c9949ad4ea None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:39:41.204989 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-f617278e-423e-4626-b3f3-29c9949ad4ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:39:41.214696 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:39:41.350101 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f617278e-423e-4626-b3f3-29c9949ad4ea None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:39:41.433767 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:39:41.467472 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1991464-dbd2-446c-b4be-2ece5dc8287c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:39:41.468001 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1991464-dbd2-446c-b4be-2ece5dc8287c service nova] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:39:41.468297 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1991464-dbd2-446c-b4be-2ece5dc8287c service nova] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:39:41.468568 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1991464-dbd2-446c-b4be-2ece5dc8287c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:39:41.470152 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-f1991464-dbd2-446c-b4be-2ece5dc8287c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:39:41.563587 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1991464-dbd2-446c-b4be-2ece5dc8287c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:39:41.565186 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1991464-dbd2-446c-b4be-2ece5dc8287c service nova] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:39:41.583741 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 729/1481] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:39:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 381 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:39:43.928226 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a37c6d-f984-40d2-a7cb-e26242d8ce73 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:39:43.928802 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a37c6d-f984-40d2-a7cb-e26242d8ce73 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:39:43.928802 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a37c6d-f984-40d2-a7cb-e26242d8ce73 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:39:43.930183 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 753/1482] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:39:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:39:43.936300 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909db6fd-90c9-4693-b157-44b3b352da42 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:39:43.936714 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909db6fd-90c9-4693-b157-44b3b352da42 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:39:43.936862 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909db6fd-90c9-4693-b157-44b3b352da42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:39:44.002939 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-909db6fd-90c9-4693-b157-44b3b352da42 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:39:44.004133 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-909db6fd-90c9-4693-b157-44b3b352da42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:39:44.050047 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:39:44.101886 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 730/1483] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:39:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:39:44.385948 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a37c6d-f984-40d2-a7cb-e26242d8ce73 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:39:44.387522 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-17a37c6d-f984-40d2-a7cb-e26242d8ce73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:39:44.411308 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:39:44.492036 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a37c6d-f984-40d2-a7cb-e26242d8ce73 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:39:44.569311 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca80071-f4df-4b14-b964-c369dd75d6b6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:39:44.569917 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca80071-f4df-4b14-b964-c369dd75d6b6 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:39:44.570197 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca80071-f4df-4b14-b964-c369dd75d6b6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:39:44.570502 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca80071-f4df-4b14-b964-c369dd75d6b6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:39:44.572120 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-5ca80071-f4df-4b14-b964-c369dd75d6b6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:39:44.649340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:39:44.691164 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca80071-f4df-4b14-b964-c369dd75d6b6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:39:44.692577 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca80071-f4df-4b14-b964-c369dd75d6b6 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:39:44.713083 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 754/1484] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:39:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:39:46.950326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91a82da-4e74-489e-97a8-6a63d85b81a1 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:39:46.950913 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91a82da-4e74-489e-97a8-6a63d85b81a1 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:39:46.950913 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f91a82da-4e74-489e-97a8-6a63d85b81a1 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:39:46.952297 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 731/1485] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:39:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:39:46.957607 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a173b7-7bf3-409e-a4f6-58aa85af204e None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:39:46.958082 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a173b7-7bf3-409e-a4f6-58aa85af204e None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:39:46.958192 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1a173b7-7bf3-409e-a4f6-58aa85af204e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:39:47.026712 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-a1a173b7-7bf3-409e-a4f6-58aa85af204e None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:39:47.027924 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a1a173b7-7bf3-409e-a4f6-58aa85af204e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:39:47.127336 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 755/1486] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:39:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:39:47.418348 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f91a82da-4e74-489e-97a8-6a63d85b81a1 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:39:47.419999 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-f91a82da-4e74-489e-97a8-6a63d85b81a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:39:47.539001 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f91a82da-4e74-489e-97a8-6a63d85b81a1 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:39:47.612143 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:39:47.613648 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f31339f-3ad8-4009-920f-3f4a576e5faa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:39:47.614266 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f31339f-3ad8-4009-920f-3f4a576e5faa service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:39:47.614816 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f31339f-3ad8-4009-920f-3f4a576e5faa service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:39:47.615306 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f31339f-3ad8-4009-920f-3f4a576e5faa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:39:47.617166 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-3f31339f-3ad8-4009-920f-3f4a576e5faa service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:39:47.730194 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f31339f-3ad8-4009-920f-3f4a576e5faa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:39:47.731572 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f31339f-3ad8-4009-920f-3f4a576e5faa service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:39:47.750776 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 732/1487] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:39:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:39:50.043063 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db69816b-ffa5-4d7d-8797-91b1d5fd4a63 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:39:50.043627 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db69816b-ffa5-4d7d-8797-91b1d5fd4a63 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:39:50.043627 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db69816b-ffa5-4d7d-8797-91b1d5fd4a63 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:39:50.045500 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 756/1488] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:39:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:39:50.050854 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5dc1e6e-ffc8-4d4e-a44a-e8013db2218c None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:39:50.051276 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5dc1e6e-ffc8-4d4e-a44a-e8013db2218c None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:39:50.051414 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5dc1e6e-ffc8-4d4e-a44a-e8013db2218c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:39:50.119367 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-f5dc1e6e-ffc8-4d4e-a44a-e8013db2218c None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:39:50.120591 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-f5dc1e6e-ffc8-4d4e-a44a-e8013db2218c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:39:50.176235 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:39:50.221040 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 733/1489] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:39:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:39:50.509225 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db69816b-ffa5-4d7d-8797-91b1d5fd4a63 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:39:50.510789 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-db69816b-ffa5-4d7d-8797-91b1d5fd4a63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:39:50.548261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:39:50.618685 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db69816b-ffa5-4d7d-8797-91b1d5fd4a63 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:39:50.693523 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955fa538-d019-4993-8515-240ba1fab180 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:39:50.694129 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955fa538-d019-4993-8515-240ba1fab180 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:39:50.694426 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955fa538-d019-4993-8515-240ba1fab180 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:39:50.694718 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955fa538-d019-4993-8515-240ba1fab180 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:39:50.696281 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-955fa538-d019-4993-8515-240ba1fab180 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:39:50.777900 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:39:50.808160 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-955fa538-d019-4993-8515-240ba1fab180 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:39:50.809583 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-955fa538-d019-4993-8515-240ba1fab180 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:39:50.827809 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 757/1490] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:39:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:39:53.210204 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e300b3-0094-4ca0-abe9-9fe22ed3d4fe None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:39:53.210825 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e300b3-0094-4ca0-abe9-9fe22ed3d4fe None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:39:53.210825 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e300b3-0094-4ca0-abe9-9fe22ed3d4fe None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:39:53.212109 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 734/1491] 10.222.0.25 () {54 vars in 927 bytes} [Fri Oct 28 09:39:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:39:53.218036 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b6aad2-b020-4f65-8089-77ed396e240d None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:39:53.218472 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b6aad2-b020-4f65-8089-77ed396e240d None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:39:53.218630 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b6aad2-b020-4f65-8089-77ed396e240d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:39:53.286129 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-d7b6aad2-b020-4f65-8089-77ed396e240d None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:39:53.287493 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d7b6aad2-b020-4f65-8089-77ed396e240d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:39:53.387671 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 758/1492] 10.222.0.25 () {58 vars in 1024 bytes} [Fri Oct 28 09:39:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:39:53.688466 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e300b3-0094-4ca0-abe9-9fe22ed3d4fe None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:39:53.690005 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-85e300b3-0094-4ca0-abe9-9fe22ed3d4fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:39:53.805155 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e300b3-0094-4ca0-abe9-9fe22ed3d4fe None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:39:53.881915 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b00c0ea-64a7-4e15-9066-34029090f43a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:39:53.882835 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b00c0ea-64a7-4e15-9066-34029090f43a service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:39:53.883113 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b00c0ea-64a7-4e15-9066-34029090f43a service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:39:53.883380 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b00c0ea-64a7-4e15-9066-34029090f43a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:39:53.885048 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-9b00c0ea-64a7-4e15-9066-34029090f43a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:39:53.894866 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:39:54.004155 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b00c0ea-64a7-4e15-9066-34029090f43a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:39:54.008321 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b00c0ea-64a7-4e15-9066-34029090f43a service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:39:54.031262 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 735/1493] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:39:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:40:03.385462 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=82 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:40:04.028880 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=50 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:41:16.918357 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8385a28b-1a85-4596-aec6-8dece0f0fc84 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:41:16.919002 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8385a28b-1a85-4596-aec6-8dece0f0fc84 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:41:16.919002 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8385a28b-1a85-4596-aec6-8dece0f0fc84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:41:17.007908 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-8385a28b-1a85-4596-aec6-8dece0f0fc84 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:41:17.009134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-8385a28b-1a85-4596-aec6-8dece0f0fc84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:41:17.130340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 759/1494] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:41:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:41:17.139794 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f898a3-de06-4b05-8238-59383d5bd5af None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:41:17.140685 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f898a3-de06-4b05-8238-59383d5bd5af None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:41:17.140685 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f898a3-de06-4b05-8238-59383d5bd5af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:41:17.228523 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-04f898a3-de06-4b05-8238-59383d5bd5af None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:41:17.229803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-04f898a3-de06-4b05-8238-59383d5bd5af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:41:17.341087 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 736/1495] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:41:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:41:17.345892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8385a28b-1a85-4596-aec6-8dece0f0fc84 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:41:17.347160 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-8385a28b-1a85-4596-aec6-8dece0f0fc84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:41:17.359159 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8385a28b-1a85-4596-aec6-8dece0f0fc84 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:41:17.438656 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:41:17.483515 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ff686c-bd6d-472d-8b40-db16063897b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:41:17.484055 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ff686c-bd6d-472d-8b40-db16063897b5 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:41:17.484339 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ff686c-bd6d-472d-8b40-db16063897b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:41:17.484618 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ff686c-bd6d-472d-8b40-db16063897b5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:41:17.485292 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ff686c-bd6d-472d-8b40-db16063897b5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:41:17.485603 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ff686c-bd6d-472d-8b40-db16063897b5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:41:17.486671 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: WARNING py.warnings [None req-04ff686c-bd6d-472d-8b40-db16063897b5 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 28 09:41:17.486671 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: warnings.warn(msg) Oct 28 09:41:17.486671 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: Oct 28 09:41:17.487597 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ff686c-bd6d-472d-8b40-db16063897b5 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:41:17.502473 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 760/1496] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:41:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:41:17.512563 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f898a3-de06-4b05-8238-59383d5bd5af None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:41:17.527263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f898a3-de06-4b05-8238-59383d5bd5af None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:41:17.602306 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:41:17.640251 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95775184-01b3-4268-9537-279d7d752723 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:41:17.640865 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95775184-01b3-4268-9537-279d7d752723 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:41:17.641145 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95775184-01b3-4268-9537-279d7d752723 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:41:17.641419 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95775184-01b3-4268-9537-279d7d752723 admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:41:17.642101 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95775184-01b3-4268-9537-279d7d752723 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:41:17.643916 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: WARNING py.warnings [None req-95775184-01b3-4268-9537-279d7d752723 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 28 09:41:17.643916 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: warnings.warn(msg) Oct 28 09:41:17.643916 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: Oct 28 09:41:17.645113 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95775184-01b3-4268-9537-279d7d752723 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:41:17.715714 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 737/1497] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:41:17 2022] POST /identity/v3/projects => generated 377 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:41:17.720289 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ff686c-bd6d-472d-8b40-db16063897b5 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:41:17.721859 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-04ff686c-bd6d-472d-8b40-db16063897b5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:41:17.761555 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ff686c-bd6d-472d-8b40-db16063897b5 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:41:17.869269 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e217a1-5b95-41ce-bcc0-a155ff86b869 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:41:17.869913 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e217a1-5b95-41ce-bcc0-a155ff86b869 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:41:17.870228 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e217a1-5b95-41ce-bcc0-a155ff86b869 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:41:17.871062 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e217a1-5b95-41ce-bcc0-a155ff86b869 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:41:17.871744 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e217a1-5b95-41ce-bcc0-a155ff86b869 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:41:17.872761 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: WARNING py.warnings [None req-86e217a1-5b95-41ce-bcc0-a155ff86b869 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 28 09:41:17.872761 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: warnings.warn(msg) Oct 28 09:41:17.872761 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: Oct 28 09:41:17.879992 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e217a1-5b95-41ce-bcc0-a155ff86b869 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:41:17.928768 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 761/1498] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:41:17 2022] POST /identity/v3/users => generated 289 bytes in 209 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:41:17.934158 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95775184-01b3-4268-9537-279d7d752723 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:41:17.935716 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-95775184-01b3-4268-9537-279d7d752723 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:41:18.012466 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95775184-01b3-4268-9537-279d7d752723 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:41:18.106348 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:41:18.122387 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8cd07e-274d-4b73-bb50-7336513a01c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:41:18.123696 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8cd07e-274d-4b73-bb50-7336513a01c7 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:41:18.124151 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8cd07e-274d-4b73-bb50-7336513a01c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:41:18.124609 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8cd07e-274d-4b73-bb50-7336513a01c7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:41:18.125587 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b8cd07e-274d-4b73-bb50-7336513a01c7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:41:18.127500 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: WARNING py.warnings [None req-9b8cd07e-274d-4b73-bb50-7336513a01c7 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 28 09:41:18.127500 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: warnings.warn(msg) Oct 28 09:41:18.127500 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: Oct 28 09:41:18.129103 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b8cd07e-274d-4b73-bb50-7336513a01c7 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:41:18.151785 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 738/1499] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:41:17 2022] GET /identity/v3/roles => generated 1420 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:41:18.158954 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e217a1-5b95-41ce-bcc0-a155ff86b869 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:41:18.161545 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-86e217a1-5b95-41ce-bcc0-a155ff86b869 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:41:18.182038 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e217a1-5b95-41ce-bcc0-a155ff86b869 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:41:18.211212 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:41:18.301546 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1575059-9f22-4908-b5a2-3aa17c242518 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:41:18.302172 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1575059-9f22-4908-b5a2-3aa17c242518 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:41:18.302491 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1575059-9f22-4908-b5a2-3aa17c242518 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:41:18.302747 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1575059-9f22-4908-b5a2-3aa17c242518 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:41:18.303453 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1575059-9f22-4908-b5a2-3aa17c242518 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:41:18.304549 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: WARNING py.warnings [None req-b1575059-9f22-4908-b5a2-3aa17c242518 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 28 09:41:18.304549 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: warnings.warn(msg) Oct 28 09:41:18.304549 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: Oct 28 09:41:18.305543 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1575059-9f22-4908-b5a2-3aa17c242518 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:41:18.320959 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 762/1500] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:41:18 2022] GET /identity/v3/roles => generated 1420 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:41:18.326673 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8cd07e-274d-4b73-bb50-7336513a01c7 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:41:18.328988 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-9b8cd07e-274d-4b73-bb50-7336513a01c7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:41:18.367462 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8cd07e-274d-4b73-bb50-7336513a01c7 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:41:18.488255 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60904091-4550-499f-8787-ec8da99ade3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:41:18.488926 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60904091-4550-499f-8787-ec8da99ade3c admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:41:18.489205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60904091-4550-499f-8787-ec8da99ade3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:41:18.489477 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60904091-4550-499f-8787-ec8da99ade3c admin admin] PATH_INFO: `/v3/projects/53d50135fb7742baab010baef2a8aa4f/users/bf51eb98b4194cbb9bf56c4150d64758/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:41:18.515134 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60904091-4550-499f-8787-ec8da99ade3c admin admin] RBAC: Authorizing `identity:create_grant(project_id=53d50135fb7742baab010baef2a8aa4f, user_id=bf51eb98b4194cbb9bf56c4150d64758, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:41:18.516450 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: WARNING py.warnings [None req-60904091-4550-499f-8787-ec8da99ade3c admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 28 09:41:18.516450 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: warnings.warn(msg) Oct 28 09:41:18.516450 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: Oct 28 09:41:18.518804 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60904091-4550-499f-8787-ec8da99ade3c admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:41:18.549164 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 739/1501] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:41:18 2022] PUT /identity/v3/projects/53d50135fb7742baab010baef2a8aa4f/users/bf51eb98b4194cbb9bf56c4150d64758/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:41:18.560918 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96358fe0-fe5e-42df-96f6-7cb2c2393956 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:41:18.561204 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96358fe0-fe5e-42df-96f6-7cb2c2393956 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:41:18.561482 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96358fe0-fe5e-42df-96f6-7cb2c2393956 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:41:18.632790 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-96358fe0-fe5e-42df-96f6-7cb2c2393956 None None] MFA Rules not processed for user `bf51eb98b4194cbb9bf56c4150d64758`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:41:18.634014 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-96358fe0-fe5e-42df-96f6-7cb2c2393956 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:41:18.721104 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 763/1502] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:41:18 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:41:18.734623 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5ec53c-f427-4311-82b1-8d3f3d46c468 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:41:18.734913 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5ec53c-f427-4311-82b1-8d3f3d46c468 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:41:18.735198 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5ec53c-f427-4311-82b1-8d3f3d46c468 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:41:18.736540 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 740/1503] 10.222.0.25 () {54 vars in 957 bytes} [Fri Oct 28 09:41:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:41:18.741585 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8416984-f011-40fa-9bab-230d35a1593a None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:41:18.741924 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8416984-f011-40fa-9bab-230d35a1593a None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:41:18.742247 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8416984-f011-40fa-9bab-230d35a1593a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:41:18.776443 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:41:18.828002 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-a8416984-f011-40fa-9bab-230d35a1593a None None] MFA Rules not processed for user `1021f1cc73b54d6eaa647a5f49815ef8`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:41:18.829242 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a8416984-f011-40fa-9bab-230d35a1593a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:41:18.910054 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 764/1504] 10.222.0.25 () {58 vars in 1054 bytes} [Fri Oct 28 09:41:18 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:41:18.918861 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e5ec53c-f427-4311-82b1-8d3f3d46c468 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:41:18.920383 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-0e5ec53c-f427-4311-82b1-8d3f3d46c468 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:41:18.977002 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:41:19.032849 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e5ec53c-f427-4311-82b1-8d3f3d46c468 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:41:19.121828 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f449d885-1a47-4b66-b007-b2c2111b43d5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:41:19.122863 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f449d885-1a47-4b66-b007-b2c2111b43d5 service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:41:19.123376 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f449d885-1a47-4b66-b007-b2c2111b43d5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:41:19.123755 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f449d885-1a47-4b66-b007-b2c2111b43d5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:41:19.125805 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-f449d885-1a47-4b66-b007-b2c2111b43d5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:41:19.240133 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f449d885-1a47-4b66-b007-b2c2111b43d5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:41:19.241619 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f449d885-1a47-4b66-b007-b2c2111b43d5 service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:41:19.245316 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:41:19.261397 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 741/1505] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:41:18 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:41:19.313158 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b4aee5-d62b-4b64-8f16-c1a12c5d251c None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:41:19.313801 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b4aee5-d62b-4b64-8f16-c1a12c5d251c None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:41:19.313937 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45b4aee5-d62b-4b64-8f16-c1a12c5d251c None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:41:19.315389 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 765/1506] 10.222.0.25 () {52 vars in 830 bytes} [Fri Oct 28 09:41:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:41:19.319818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f449d885-1a47-4b66-b007-b2c2111b43d5 service cinder] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:41:19.321057 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-f449d885-1a47-4b66-b007-b2c2111b43d5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:41:19.451323 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f449d885-1a47-4b66-b007-b2c2111b43d5 service cinder] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:41:19.557572 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d708695c-a206-469c-827f-a18528848c29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:41:19.558794 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d708695c-a206-469c-827f-a18528848c29 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:41:19.559066 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d708695c-a206-469c-827f-a18528848c29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:41:19.559338 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d708695c-a206-469c-827f-a18528848c29 admin admin] PATH_INFO: `/v3/users/bf51eb98b4194cbb9bf56c4150d64758` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:41:19.573508 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d708695c-a206-469c-827f-a18528848c29 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bf51eb98b4194cbb9bf56c4150d64758)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:41:19.574608 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: WARNING py.warnings [None req-d708695c-a206-469c-827f-a18528848c29 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 28 09:41:19.574608 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: warnings.warn(msg) Oct 28 09:41:19.574608 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: Oct 28 09:41:19.575875 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d708695c-a206-469c-827f-a18528848c29 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:41:19.586344 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:41:19.649474 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-d708695c-a206-469c-827f-a18528848c29 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bf51eb98b4194cbb9bf56c4150d64758'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:41:19.659181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-d708695c-a206-469c-827f-a18528848c29 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bf51eb98b4194cbb9bf56c4150d64758'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:41:19.659791 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-d708695c-a206-469c-827f-a18528848c29 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bf51eb98b4194cbb9bf56c4150d64758'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:41:19.660262 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-d708695c-a206-469c-827f-a18528848c29 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bf51eb98b4194cbb9bf56c4150d64758'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:41:19.676142 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-d708695c-a206-469c-827f-a18528848c29 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bf51eb98b4194cbb9bf56c4150d64758'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:41:19.715534 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 742/1507] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:41:19 2022] DELETE /identity/v3/users/bf51eb98b4194cbb9bf56c4150d64758 => generated 0 bytes in 396 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:41:19.720161 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da24a057-9b06-446a-8384-5b4a146d4b91 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:41:19.720486 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da24a057-9b06-446a-8384-5b4a146d4b91 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:41:19.720789 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da24a057-9b06-446a-8384-5b4a146d4b91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:41:19.789384 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-da24a057-9b06-446a-8384-5b4a146d4b91 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:41:19.790892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-da24a057-9b06-446a-8384-5b4a146d4b91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:41:19.892823 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:41:19.915086 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 766/1508] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:41:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:41:19.925445 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d708695c-a206-469c-827f-a18528848c29 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:41:19.926995 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-d708695c-a206-469c-827f-a18528848c29 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:41:20.058311 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d708695c-a206-469c-827f-a18528848c29 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:41:20.141695 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:41:20.168512 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f0f0c83-9c3b-485e-add4-eb16a3999fa5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:41:20.169144 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f0f0c83-9c3b-485e-add4-eb16a3999fa5 service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:41:20.169425 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f0f0c83-9c3b-485e-add4-eb16a3999fa5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:41:20.169696 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f0f0c83-9c3b-485e-add4-eb16a3999fa5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:41:20.171382 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-7f0f0c83-9c3b-485e-add4-eb16a3999fa5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:41:20.355477 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f0f0c83-9c3b-485e-add4-eb16a3999fa5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:41:20.358256 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f0f0c83-9c3b-485e-add4-eb16a3999fa5 service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:41:20.393487 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 743/1509] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:41:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 469 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:41:20.805226 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da24a057-9b06-446a-8384-5b4a146d4b91 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:41:20.807468 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-da24a057-9b06-446a-8384-5b4a146d4b91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:41:20.829243 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da24a057-9b06-446a-8384-5b4a146d4b91 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:41:20.944845 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd0a6f9-99c8-447b-9c02-a758b041b983 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:41:20.945469 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd0a6f9-99c8-447b-9c02-a758b041b983 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:41:20.945792 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd0a6f9-99c8-447b-9c02-a758b041b983 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:41:20.946103 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd0a6f9-99c8-447b-9c02-a758b041b983 admin admin] PATH_INFO: `/v3/projects/53d50135fb7742baab010baef2a8aa4f` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:41:20.957051 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd0a6f9-99c8-447b-9c02-a758b041b983 admin admin] RBAC: Authorizing `identity:delete_project(project_id=53d50135fb7742baab010baef2a8aa4f)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:41:20.958161 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: WARNING py.warnings [None req-1fd0a6f9-99c8-447b-9c02-a758b041b983 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 28 09:41:20.958161 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: warnings.warn(msg) Oct 28 09:41:20.958161 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: Oct 28 09:41:20.959575 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd0a6f9-99c8-447b-9c02-a758b041b983 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:41:21.025347 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-1fd0a6f9-99c8-447b-9c02-a758b041b983 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '53d50135fb7742baab010baef2a8aa4f', 'request_id': 'req-1fd0a6f9-99c8-447b-9c02-a758b041b983'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:41:21.051667 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-1fd0a6f9-99c8-447b-9c02-a758b041b983 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '53d50135fb7742baab010baef2a8aa4f', 'request_id': 'req-1fd0a6f9-99c8-447b-9c02-a758b041b983'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:41:21.066141 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-1fd0a6f9-99c8-447b-9c02-a758b041b983 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '53d50135fb7742baab010baef2a8aa4f', 'request_id': 'req-1fd0a6f9-99c8-447b-9c02-a758b041b983'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:41:21.107520 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-1fd0a6f9-99c8-447b-9c02-a758b041b983 admin admin] The token cache is being invalidate because project 53d50135fb7742baab010baef2a8aa4f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:41:21.108002 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-1fd0a6f9-99c8-447b-9c02-a758b041b983 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:41:21.109519 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-1fd0a6f9-99c8-447b-9c02-a758b041b983 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:41:21.111890 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 767/1510] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:41:20 2022] DELETE /identity/v3/projects/53d50135fb7742baab010baef2a8aa4f => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:41:30.388938 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=62 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:41:31.103318 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:42:53.996954 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8932590-8d4f-4ac4-9378-3bf2be5b64f7 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:42:53.997561 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8932590-8d4f-4ac4-9378-3bf2be5b64f7 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:42:53.997561 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8932590-8d4f-4ac4-9378-3bf2be5b64f7 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:42:53.998866 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 744/1511] 127.0.0.1 () {54 vars in 901 bytes} [Fri Oct 28 09:42:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:42:54.003906 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e01fa95-e5f7-4c89-8b53-1e51b0f08826 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:42:54.004324 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e01fa95-e5f7-4c89-8b53-1e51b0f08826 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:42:54.004445 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e01fa95-e5f7-4c89-8b53-1e51b0f08826 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:42:54.093032 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-2e01fa95-e5f7-4c89-8b53-1e51b0f08826 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:42:54.094328 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-2e01fa95-e5f7-4c89-8b53-1e51b0f08826 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:42:54.203246 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 768/1512] 127.0.0.1 () {58 vars in 998 bytes} [Fri Oct 28 09:42:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:42:54.357277 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8932590-8d4f-4ac4-9378-3bf2be5b64f7 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:42:54.359439 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-d8932590-8d4f-4ac4-9378-3bf2be5b64f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:42:54.480721 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8932590-8d4f-4ac4-9378-3bf2be5b64f7 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:42:54.486286 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e13785-5e57-4810-9f3e-f325cf7a0a40 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:42:54.486927 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e13785-5e57-4810-9f3e-f325cf7a0a40 service nova] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:42:54.487213 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e13785-5e57-4810-9f3e-f325cf7a0a40 service nova] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:42:54.487495 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e13785-5e57-4810-9f3e-f325cf7a0a40 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:42:54.496631 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e13785-5e57-4810-9f3e-f325cf7a0a40 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:42:54.498227 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e13785-5e57-4810-9f3e-f325cf7a0a40 service nova] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:42:54.507701 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 745/1513] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:42:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:42:55.418088 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f244fb9-26ce-413f-be22-e13ef18539c1 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:42:55.418669 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f244fb9-26ce-413f-be22-e13ef18539c1 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:42:55.418669 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f244fb9-26ce-413f-be22-e13ef18539c1 None None] PATH_INFO: `/v3` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:42:55.419939 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 769/1514] 10.222.0.55 () {54 vars in 925 bytes} [Fri Oct 28 09:42:55 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:42:55.432349 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760cd4e8-bbd8-4c5b-aae5-9a38f46de353 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:42:55.432836 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760cd4e8-bbd8-4c5b-aae5-9a38f46de353 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:42:55.432961 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760cd4e8-bbd8-4c5b-aae5-9a38f46de353 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:42:55.495518 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-760cd4e8-bbd8-4c5b-aae5-9a38f46de353 None None] MFA Rules not processed for user `3f1d95049d2d4593bf52670b1a0035fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:42:55.496736 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-760cd4e8-bbd8-4c5b-aae5-9a38f46de353 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:42:55.575517 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:42:55.582643 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 746/1515] 10.222.0.55 () {58 vars in 1015 bytes} [Fri Oct 28 09:42:55 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:42:55.604500 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812dd4ab-2479-40c3-b2e6-3fd3d1b72ce2 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:42:55.604789 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812dd4ab-2479-40c3-b2e6-3fd3d1b72ce2 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:42:55.605070 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812dd4ab-2479-40c3-b2e6-3fd3d1b72ce2 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:42:55.606423 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 770/1516] 127.0.0.1 () {54 vars in 901 bytes} [Fri Oct 28 09:42:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:42:55.612651 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d943b9d-0dbc-4152-926e-fb925763e58f None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:42:55.612942 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d943b9d-0dbc-4152-926e-fb925763e58f None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:42:55.613229 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d943b9d-0dbc-4152-926e-fb925763e58f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:42:55.633367 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-5d943b9d-0dbc-4152-926e-fb925763e58f None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:42:55.635182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-5d943b9d-0dbc-4152-926e-fb925763e58f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:42:55.643110 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 747/1517] 127.0.0.1 () {58 vars in 998 bytes} [Fri Oct 28 09:42:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:42:55.793685 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-812dd4ab-2479-40c3-b2e6-3fd3d1b72ce2 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:42:55.807885 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-812dd4ab-2479-40c3-b2e6-3fd3d1b72ce2 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:42:55.813389 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e287cf06-8e9a-4607-8c26-a1b5ea5bf423 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:42:55.814028 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e287cf06-8e9a-4607-8c26-a1b5ea5bf423 service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:42:55.814311 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e287cf06-8e9a-4607-8c26-a1b5ea5bf423 service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:42:55.814615 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e287cf06-8e9a-4607-8c26-a1b5ea5bf423 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:42:55.823500 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e287cf06-8e9a-4607-8c26-a1b5ea5bf423 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:42:55.825090 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e287cf06-8e9a-4607-8c26-a1b5ea5bf423 service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:42:55.835000 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 771/1518] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:42:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:43:04.317979 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494d18aa-3af6-42cb-840e-77a512c4b151 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:43:04.318496 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494d18aa-3af6-42cb-840e-77a512c4b151 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:43:04.318589 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494d18aa-3af6-42cb-840e-77a512c4b151 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:43:04.319895 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 748/1519] 10.222.0.25 () {54 vars in 961 bytes} [Fri Oct 28 09:43:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:43:04.325984 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eacc931a-ccc0-4ec3-af74-74c4fb8d04e9 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:43:04.326541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eacc931a-ccc0-4ec3-af74-74c4fb8d04e9 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:43:04.327094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eacc931a-ccc0-4ec3-af74-74c4fb8d04e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:43:04.348093 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-eacc931a-ccc0-4ec3-af74-74c4fb8d04e9 None None] MFA Rules not processed for user `3f1d95049d2d4593bf52670b1a0035fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:43:04.349313 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-eacc931a-ccc0-4ec3-af74-74c4fb8d04e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:43:04.358569 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 772/1520] 10.222.0.25 () {58 vars in 1058 bytes} [Fri Oct 28 09:43:04 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:43:04.366347 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-494d18aa-3af6-42cb-840e-77a512c4b151 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:43:04.380944 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-494d18aa-3af6-42cb-840e-77a512c4b151 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:43:04.385928 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2101600b-f7c2-4844-92e2-d3bc48ccb1db service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3f1d95049d2d4593bf52670b1a0035fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:43:04.386575 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2101600b-f7c2-4844-92e2-d3bc48ccb1db service placement] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:43:04.386874 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2101600b-f7c2-4844-92e2-d3bc48ccb1db service placement] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:43:04.387129 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2101600b-f7c2-4844-92e2-d3bc48ccb1db service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:43:04.395500 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2101600b-f7c2-4844-92e2-d3bc48ccb1db service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:43:04.397078 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2101600b-f7c2-4844-92e2-d3bc48ccb1db service placement] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:43:04.407485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 749/1521] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:43:04 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:43:07.337768 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0768e7-fe6c-497d-99ee-e3ddf34d1699 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:43:07.338328 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0768e7-fe6c-497d-99ee-e3ddf34d1699 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:43:07.338328 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0768e7-fe6c-497d-99ee-e3ddf34d1699 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:43:07.339699 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 773/1522] 127.0.0.1 () {54 vars in 901 bytes} [Fri Oct 28 09:43:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:43:07.346183 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d202f38-4c8b-4b3e-81ce-366a8d5f2d46 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:43:07.346770 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d202f38-4c8b-4b3e-81ce-366a8d5f2d46 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:43:07.346770 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d202f38-4c8b-4b3e-81ce-366a8d5f2d46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:43:07.366729 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-6d202f38-4c8b-4b3e-81ce-366a8d5f2d46 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:43:07.368477 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-6d202f38-4c8b-4b3e-81ce-366a8d5f2d46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:43:07.376333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 750/1523] 127.0.0.1 () {58 vars in 998 bytes} [Fri Oct 28 09:43:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:43:07.532259 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0768e7-fe6c-497d-99ee-e3ddf34d1699 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:43:07.548521 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0768e7-fe6c-497d-99ee-e3ddf34d1699 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:43:07.554039 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e84006ea-21c5-4719-95e4-c5e569eb5e9f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:43:07.554694 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84006ea-21c5-4719-95e4-c5e569eb5e9f service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:43:07.554980 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84006ea-21c5-4719-95e4-c5e569eb5e9f service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:43:07.555259 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84006ea-21c5-4719-95e4-c5e569eb5e9f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:43:07.564876 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e84006ea-21c5-4719-95e4-c5e569eb5e9f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:43:07.566501 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e84006ea-21c5-4719-95e4-c5e569eb5e9f service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:43:07.576856 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 774/1524] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:43:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:43:08.662305 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c3ace5-6e92-4609-8432-3c05a2189297 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:43:08.662931 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c3ace5-6e92-4609-8432-3c05a2189297 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:43:08.662931 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c3ace5-6e92-4609-8432-3c05a2189297 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:43:08.664223 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 751/1525] 127.0.0.1 () {54 vars in 901 bytes} [Fri Oct 28 09:43:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:43:08.671511 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b113a57-7c5a-4fc7-a72c-4db4062ac371 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:43:08.671987 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b113a57-7c5a-4fc7-a72c-4db4062ac371 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:43:08.672085 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b113a57-7c5a-4fc7-a72c-4db4062ac371 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:43:08.692098 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-3b113a57-7c5a-4fc7-a72c-4db4062ac371 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:43:08.693316 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-3b113a57-7c5a-4fc7-a72c-4db4062ac371 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:43:08.701371 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 775/1526] 127.0.0.1 () {58 vars in 998 bytes} [Fri Oct 28 09:43:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:43:08.850310 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c3ace5-6e92-4609-8432-3c05a2189297 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:43:08.865068 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c3ace5-6e92-4609-8432-3c05a2189297 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:43:08.871198 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6aeabda-9181-4a4e-83fd-ff4d5ed1d7da service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:43:08.871795 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6aeabda-9181-4a4e-83fd-ff4d5ed1d7da service nova] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:43:08.872079 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6aeabda-9181-4a4e-83fd-ff4d5ed1d7da service nova] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:43:08.872358 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6aeabda-9181-4a4e-83fd-ff4d5ed1d7da service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:43:08.880603 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6aeabda-9181-4a4e-83fd-ff4d5ed1d7da service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:43:08.882190 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6aeabda-9181-4a4e-83fd-ff4d5ed1d7da service nova] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:43:08.891908 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 752/1527] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:43:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:43:13.085490 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c72e65-c06d-4669-9684-7fc04efc4394 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:43:13.086060 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c72e65-c06d-4669-9684-7fc04efc4394 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:43:13.086060 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c72e65-c06d-4669-9684-7fc04efc4394 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:43:13.087487 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 776/1528] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 09:43:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:43:13.093651 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b29527-837e-429b-9e61-17597251f36c None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:43:13.094138 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b29527-837e-429b-9e61-17597251f36c None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:43:13.094230 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b29527-837e-429b-9e61-17597251f36c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:43:13.113640 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-24b29527-837e-429b-9e61-17597251f36c None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:43:13.114865 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-24b29527-837e-429b-9e61-17597251f36c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:43:13.122876 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 753/1529] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 09:43:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:43:13.406417 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c72e65-c06d-4669-9684-7fc04efc4394 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:43:13.408179 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-24c72e65-c06d-4669-9684-7fc04efc4394 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:43:13.443401 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:43:13.494564 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c72e65-c06d-4669-9684-7fc04efc4394 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:43:13.499842 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def7e96b-9727-4453-9d8c-bc5b863dcd2d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:43:13.500465 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def7e96b-9727-4453-9d8c-bc5b863dcd2d service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:43:13.500746 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def7e96b-9727-4453-9d8c-bc5b863dcd2d service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:43:13.501020 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def7e96b-9727-4453-9d8c-bc5b863dcd2d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:43:13.511096 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-def7e96b-9727-4453-9d8c-bc5b863dcd2d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:43:13.512521 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-def7e96b-9727-4453-9d8c-bc5b863dcd2d service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:43:13.522204 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 777/1530] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:43:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:43:15.695880 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0f8f8b-9431-4de0-93b9-d5781a71338d None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:43:15.696441 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0f8f8b-9431-4de0-93b9-d5781a71338d None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:43:15.696441 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0f8f8b-9431-4de0-93b9-d5781a71338d None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:43:15.697785 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 754/1531] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 09:43:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 09:43:15.703431 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07dbb75f-597b-4936-8f62-af8666ea3c21 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:43:15.703871 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07dbb75f-597b-4936-8f62-af8666ea3c21 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:43:15.703969 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07dbb75f-597b-4936-8f62-af8666ea3c21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:43:15.723558 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-07dbb75f-597b-4936-8f62-af8666ea3c21 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:43:15.724756 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-07dbb75f-597b-4936-8f62-af8666ea3c21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:43:15.733181 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 778/1532] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 09:43:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:43:16.016430 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0f8f8b-9431-4de0-93b9-d5781a71338d None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:43:16.027977 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0f8f8b-9431-4de0-93b9-d5781a71338d None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:43:16.031544 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11cdb990-9265-45a8-9d97-462696bbedd9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:43:16.032144 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cdb990-9265-45a8-9d97-462696bbedd9 service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:43:16.032419 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cdb990-9265-45a8-9d97-462696bbedd9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:43:16.032690 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11cdb990-9265-45a8-9d97-462696bbedd9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:43:16.043167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11cdb990-9265-45a8-9d97-462696bbedd9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:43:16.044599 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11cdb990-9265-45a8-9d97-462696bbedd9 service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:43:16.054293 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 755/1533] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:43:16 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:43:20.026988 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a08696-76f1-4449-a106-435b16d5ef67 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:43:20.027527 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a08696-76f1-4449-a106-435b16d5ef67 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:43:20.027527 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a08696-76f1-4449-a106-435b16d5ef67 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:43:20.028889 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 779/1534] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 09:43:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:43:20.034124 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2bcd47d-908e-453c-8ca3-fcdc59925f3d None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:43:20.034594 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2bcd47d-908e-453c-8ca3-fcdc59925f3d None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:43:20.034690 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2bcd47d-908e-453c-8ca3-fcdc59925f3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:43:20.054888 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-f2bcd47d-908e-453c-8ca3-fcdc59925f3d None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:43:20.056763 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-f2bcd47d-908e-453c-8ca3-fcdc59925f3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:43:20.065017 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 756/1535] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 09:43:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:43:20.368047 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a08696-76f1-4449-a106-435b16d5ef67 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:43:20.369919 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-f7a08696-76f1-4449-a106-435b16d5ef67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:43:20.456889 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a08696-76f1-4449-a106-435b16d5ef67 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:43:20.460942 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c01626c-7773-466b-b682-a01eccf5870d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:43:20.462110 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c01626c-7773-466b-b682-a01eccf5870d service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:43:20.462420 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c01626c-7773-466b-b682-a01eccf5870d service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:43:20.462693 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c01626c-7773-466b-b682-a01eccf5870d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:43:20.473267 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c01626c-7773-466b-b682-a01eccf5870d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:43:20.474767 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c01626c-7773-466b-b682-a01eccf5870d service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:43:20.486382 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 780/1536] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:43:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:43:22.720056 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fb3859-191a-4ca8-9f72-a73d0cefb52c None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:43:22.720656 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fb3859-191a-4ca8-9f72-a73d0cefb52c None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:43:22.720656 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fb3859-191a-4ca8-9f72-a73d0cefb52c None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:43:22.721923 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 757/1537] 127.0.0.1 () {54 vars in 917 bytes} [Fri Oct 28 09:43:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:43:22.728103 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c8283c-180d-48d6-92e7-615846b0ccb8 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:43:22.728574 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c8283c-180d-48d6-92e7-615846b0ccb8 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:43:22.728669 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c8283c-180d-48d6-92e7-615846b0ccb8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:43:22.748286 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-26c8283c-180d-48d6-92e7-615846b0ccb8 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:43:22.749508 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-26c8283c-180d-48d6-92e7-615846b0ccb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:43:22.757566 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 781/1538] 127.0.0.1 () {58 vars in 1014 bytes} [Fri Oct 28 09:43:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:43:23.060518 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3fb3859-191a-4ca8-9f72-a73d0cefb52c None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:43:23.062195 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-c3fb3859-191a-4ca8-9f72-a73d0cefb52c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:43:23.076241 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3fb3859-191a-4ca8-9f72-a73d0cefb52c None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:43:23.080241 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95fc56c6-8aa0-4c1d-8d9b-ef7087bfcad5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:43:23.081042 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95fc56c6-8aa0-4c1d-8d9b-ef7087bfcad5 service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:43:23.081324 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95fc56c6-8aa0-4c1d-8d9b-ef7087bfcad5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:43:23.081604 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95fc56c6-8aa0-4c1d-8d9b-ef7087bfcad5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:43:23.092560 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95fc56c6-8aa0-4c1d-8d9b-ef7087bfcad5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:43:23.093994 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95fc56c6-8aa0-4c1d-8d9b-ef7087bfcad5 service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:43:23.105391 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 758/1539] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:43:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:43:32.743429 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=63 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:43:33.099991 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=37 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:44:50.346555 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8537345a-13e4-4cfa-8194-d6b04c8223d1 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:50.347632 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8537345a-13e4-4cfa-8194-d6b04c8223d1 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:50.347632 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8537345a-13e4-4cfa-8194-d6b04c8223d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:50.373204 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-8537345a-13e4-4cfa-8194-d6b04c8223d1 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:44:50.374492 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-8537345a-13e4-4cfa-8194-d6b04c8223d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:44:50.390026 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 782/1540] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:44:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:44:50.404556 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae007ba6-47da-42d8-9b0f-3e1d1d433b72 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:50.405679 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae007ba6-47da-42d8-9b0f-3e1d1d433b72 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:50.406188 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae007ba6-47da-42d8-9b0f-3e1d1d433b72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:50.434919 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-ae007ba6-47da-42d8-9b0f-3e1d1d433b72 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:44:50.435612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95555f53-6882-4467-8a6e-d02d16200de8 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:50.436569 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95555f53-6882-4467-8a6e-d02d16200de8 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:50.436569 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95555f53-6882-4467-8a6e-d02d16200de8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:50.437288 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-ae007ba6-47da-42d8-9b0f-3e1d1d433b72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:44:50.453293 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 759/1541] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:44:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:44:50.461398 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-95555f53-6882-4467-8a6e-d02d16200de8 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:44:50.462692 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-95555f53-6882-4467-8a6e-d02d16200de8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:44:50.469294 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae007ba6-47da-42d8-9b0f-3e1d1d433b72 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:50.480902 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 783/1542] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:44:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:44:50.482134 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae007ba6-47da-42d8-9b0f-3e1d1d433b72 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:50.486480 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbaa691-2524-4aec-8060-0376df354c6b None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:50.486480 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbaa691-2524-4aec-8060-0376df354c6b None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:50.486480 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbaa691-2524-4aec-8060-0376df354c6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:50.500192 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b4677d5-9b0c-4d50-afc8-5cdf6bb9e63f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:50.500981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4677d5-9b0c-4d50-afc8-5cdf6bb9e63f admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:50.501364 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4677d5-9b0c-4d50-afc8-5cdf6bb9e63f admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:50.501708 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4677d5-9b0c-4d50-afc8-5cdf6bb9e63f admin admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:50.502595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b4677d5-9b0c-4d50-afc8-5cdf6bb9e63f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:44:50.503091 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b4677d5-9b0c-4d50-afc8-5cdf6bb9e63f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:50.504204 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: WARNING py.warnings [None req-8b4677d5-9b0c-4d50-afc8-5cdf6bb9e63f admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Oct 28 09:44:50.504204 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: warnings.warn(msg) Oct 28 09:44:50.504204 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: Oct 28 09:44:50.505281 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b4677d5-9b0c-4d50-afc8-5cdf6bb9e63f admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:50.513160 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-3fbaa691-2524-4aec-8060-0376df354c6b None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:44:50.514388 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-3fbaa691-2524-4aec-8060-0376df354c6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:44:50.531602 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 760/1543] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:44:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:44:50.532945 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271e6242-1759-4375-b9de-837a2fa110a9 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:50.533076 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 784/1544] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:44:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:44:50.533252 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271e6242-1759-4375-b9de-837a2fa110a9 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:50.533514 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271e6242-1759-4375-b9de-837a2fa110a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:50.534295 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359611c0-c605-4916-960d-658807c94bdb None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:50.534614 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359611c0-c605-4916-960d-658807c94bdb None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:50.534894 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359611c0-c605-4916-960d-658807c94bdb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:50.563571 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-359611c0-c605-4916-960d-658807c94bdb None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:44:50.564833 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-359611c0-c605-4916-960d-658807c94bdb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:44:50.583691 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-271e6242-1759-4375-b9de-837a2fa110a9 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:44:50.585336 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-271e6242-1759-4375-b9de-837a2fa110a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:44:50.592577 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 785/1545] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:44:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 59 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:44:50.594049 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4808f3b9-a140-4775-b49c-fe49b3fecb28 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:50.594346 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4808f3b9-a140-4775-b49c-fe49b3fecb28 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:50.594652 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4808f3b9-a140-4775-b49c-fe49b3fecb28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:50.607048 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 761/1546] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:44:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:44:50.607815 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-271e6242-1759-4375-b9de-837a2fa110a9 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:50.625894 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-271e6242-1759-4375-b9de-837a2fa110a9 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:50.638252 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b229e0b-97ca-47a4-91b8-a02e3e89a306 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:50.638508 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-4808f3b9-a140-4775-b49c-fe49b3fecb28 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:44:50.638926 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b229e0b-97ca-47a4-91b8-a02e3e89a306 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:50.639207 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b229e0b-97ca-47a4-91b8-a02e3e89a306 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:50.639476 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b229e0b-97ca-47a4-91b8-a02e3e89a306 admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:50.639647 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4808f3b9-a140-4775-b49c-fe49b3fecb28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:44:50.640115 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b229e0b-97ca-47a4-91b8-a02e3e89a306 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:50.645399 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b229e0b-97ca-47a4-91b8-a02e3e89a306 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:50.663639 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 786/1547] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:44:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 70 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:44:50.664916 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb4fd4a-ea4a-4127-a990-73a235d819e3 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:50.665198 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb4fd4a-ea4a-4127-a990-73a235d819e3 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:50.665716 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb4fd4a-ea4a-4127-a990-73a235d819e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:50.697164 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-7cb4fd4a-ea4a-4127-a990-73a235d819e3 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:44:50.698725 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7cb4fd4a-ea4a-4127-a990-73a235d819e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:44:50.705598 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 762/1548] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:44:50 2022] POST /identity/v3/projects => generated 393 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:44:50.706801 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b229e0b-97ca-47a4-91b8-a02e3e89a306 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:50.707047 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 787/1549] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:44:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:44:50.708346 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122bb792-a97d-4023-bc79-094b0affa80c None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:50.708937 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122bb792-a97d-4023-bc79-094b0affa80c None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:50.709107 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122bb792-a97d-4023-bc79-094b0affa80c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:50.730469 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-122bb792-a97d-4023-bc79-094b0affa80c None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:44:50.731116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-122bb792-a97d-4023-bc79-094b0affa80c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:44:50.776474 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b229e0b-97ca-47a4-91b8-a02e3e89a306 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:50.781218 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b44a185-cde1-4471-a445-ee94e31a972d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:50.781830 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b44a185-cde1-4471-a445-ee94e31a972d admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:50.782117 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b44a185-cde1-4471-a445-ee94e31a972d admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:50.782523 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b44a185-cde1-4471-a445-ee94e31a972d admin admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:50.783066 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b44a185-cde1-4471-a445-ee94e31a972d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:44:50.783374 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b44a185-cde1-4471-a445-ee94e31a972d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:50.784977 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b44a185-cde1-4471-a445-ee94e31a972d admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:50.795125 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 788/1550] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:44:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:44:50.795780 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122bb792-a97d-4023-bc79-094b0affa80c None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:50.805627 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122bb792-a97d-4023-bc79-094b0affa80c None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:50.806100 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 763/1551] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:44:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:44:50.807445 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3b3242-b7b7-4514-bbb2-653c4b38dfeb None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:50.807707 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3b3242-b7b7-4514-bbb2-653c4b38dfeb None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:50.807993 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3b3242-b7b7-4514-bbb2-653c4b38dfeb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:50.810350 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebcdbdea-86b7-428f-9f7d-9d98da61e1fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:50.810985 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebcdbdea-86b7-428f-9f7d-9d98da61e1fa admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:50.811263 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebcdbdea-86b7-428f-9f7d-9d98da61e1fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:50.811536 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebcdbdea-86b7-428f-9f7d-9d98da61e1fa admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:50.812177 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebcdbdea-86b7-428f-9f7d-9d98da61e1fa admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:44:50.812483 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebcdbdea-86b7-428f-9f7d-9d98da61e1fa admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:50.814057 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebcdbdea-86b7-428f-9f7d-9d98da61e1fa admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:50.826728 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 789/1552] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:44:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:44:50.827946 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060c7978-c5e1-4c97-a5d0-e4d8eb3677bd None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:50.828232 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060c7978-c5e1-4c97-a5d0-e4d8eb3677bd None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:50.828350 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-fc3b3242-b7b7-4514-bbb2-653c4b38dfeb None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:44:50.828534 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060c7978-c5e1-4c97-a5d0-e4d8eb3677bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:50.829516 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-fc3b3242-b7b7-4514-bbb2-653c4b38dfeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:44:50.837646 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 764/1553] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:44:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:44:50.842671 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3b3242-b7b7-4514-bbb2-653c4b38dfeb None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:50.848021 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-060c7978-c5e1-4c97-a5d0-e4d8eb3677bd None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:44:50.849219 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-060c7978-c5e1-4c97-a5d0-e4d8eb3677bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:44:50.852215 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3b3242-b7b7-4514-bbb2-653c4b38dfeb None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:50.857035 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c08d42f-b8ca-4c95-a5d6-bc1ba593c6cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:50.857702 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c08d42f-b8ca-4c95-a5d6-bc1ba593c6cc admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:50.857842 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 790/1554] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:44:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:44:50.857930 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c08d42f-b8ca-4c95-a5d6-bc1ba593c6cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:50.858184 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c08d42f-b8ca-4c95-a5d6-bc1ba593c6cc admin admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:50.858505 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-060c7978-c5e1-4c97-a5d0-e4d8eb3677bd None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:50.858905 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c08d42f-b8ca-4c95-a5d6-bc1ba593c6cc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:50.859865 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: WARNING py.warnings [None req-0c08d42f-b8ca-4c95-a5d6-bc1ba593c6cc admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 28 09:44:50.859865 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: warnings.warn(msg) Oct 28 09:44:50.859865 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: Oct 28 09:44:50.861122 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c08d42f-b8ca-4c95-a5d6-bc1ba593c6cc admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:50.868402 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-060c7978-c5e1-4c97-a5d0-e4d8eb3677bd None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:50.873309 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5300d42d-2981-403e-8fdc-dea20ae686f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:50.873919 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5300d42d-2981-403e-8fdc-dea20ae686f0 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:50.874251 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5300d42d-2981-403e-8fdc-dea20ae686f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:50.874575 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5300d42d-2981-403e-8fdc-dea20ae686f0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:50.875225 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5300d42d-2981-403e-8fdc-dea20ae686f0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:44:50.875720 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5300d42d-2981-403e-8fdc-dea20ae686f0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:50.877350 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5300d42d-2981-403e-8fdc-dea20ae686f0 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:50.885919 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 765/1555] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:44:50 2022] POST /identity/v3/users => generated 297 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:44:50.887100 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6711866b-07f9-48d9-8578-1a0df40dd6e8 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:50.887492 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6711866b-07f9-48d9-8578-1a0df40dd6e8 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:50.887774 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6711866b-07f9-48d9-8578-1a0df40dd6e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:50.900049 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 791/1556] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:44:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:44:50.900049 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afccb892-6552-4c02-8cf0-3e2e3a3745cb None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:50.900049 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afccb892-6552-4c02-8cf0-3e2e3a3745cb None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:50.900049 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afccb892-6552-4c02-8cf0-3e2e3a3745cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:50.910265 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-6711866b-07f9-48d9-8578-1a0df40dd6e8 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:44:50.911493 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-6711866b-07f9-48d9-8578-1a0df40dd6e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:44:50.917390 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-afccb892-6552-4c02-8cf0-3e2e3a3745cb None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:44:50.918595 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-afccb892-6552-4c02-8cf0-3e2e3a3745cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:44:50.920126 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 766/1557] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:44:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:44:50.920906 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6711866b-07f9-48d9-8578-1a0df40dd6e8 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:50.927154 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 792/1558] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:44:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:44:50.928057 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afccb892-6552-4c02-8cf0-3e2e3a3745cb None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:50.934981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6711866b-07f9-48d9-8578-1a0df40dd6e8 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:50.938187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afccb892-6552-4c02-8cf0-3e2e3a3745cb None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:50.939554 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c93ba21c-a915-441f-ab50-82e9b394fcc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:50.940156 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93ba21c-a915-441f-ab50-82e9b394fcc8 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:50.940581 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93ba21c-a915-441f-ab50-82e9b394fcc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:50.940855 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c93ba21c-a915-441f-ab50-82e9b394fcc8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:50.941514 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c93ba21c-a915-441f-ab50-82e9b394fcc8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:50.942749 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89276f70-b249-4e0e-b3fa-afbf993c2dd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:50.943337 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89276f70-b249-4e0e-b3fa-afbf993c2dd2 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:50.943623 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89276f70-b249-4e0e-b3fa-afbf993c2dd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:50.943902 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89276f70-b249-4e0e-b3fa-afbf993c2dd2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:50.944077 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c93ba21c-a915-441f-ab50-82e9b394fcc8 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:50.944615 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89276f70-b249-4e0e-b3fa-afbf993c2dd2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:50.945646 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: WARNING py.warnings [None req-89276f70-b249-4e0e-b3fa-afbf993c2dd2 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 28 09:44:50.945646 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: warnings.warn(msg) Oct 28 09:44:50.945646 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: Oct 28 09:44:50.946857 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89276f70-b249-4e0e-b3fa-afbf993c2dd2 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:50.974188 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 767/1559] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:44:50 2022] POST /identity/v3/projects => generated 377 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:44:50.974188 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75030ed8-9501-4309-be12-ee35f93e300f None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:50.974188 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75030ed8-9501-4309-be12-ee35f93e300f None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:50.974188 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75030ed8-9501-4309-be12-ee35f93e300f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:50.979534 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 793/1560] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:44:50 2022] POST /identity/v3/projects => generated 385 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:44:50.980869 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46704303-2e5c-4d0a-9b62-a79d19794f7a None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:50.981163 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46704303-2e5c-4d0a-9b62-a79d19794f7a None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:50.981444 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46704303-2e5c-4d0a-9b62-a79d19794f7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:50.996222 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-75030ed8-9501-4309-be12-ee35f93e300f None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:44:50.997419 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-75030ed8-9501-4309-be12-ee35f93e300f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:44:51.001895 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-46704303-2e5c-4d0a-9b62-a79d19794f7a None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:44:51.003090 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-46704303-2e5c-4d0a-9b62-a79d19794f7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:44:51.038145 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:44:51.057735 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:44:51.060664 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 768/1561] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:44:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:44:51.065205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75030ed8-9501-4309-be12-ee35f93e300f None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:51.067716 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 794/1562] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:44:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:44:51.067716 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46704303-2e5c-4d0a-9b62-a79d19794f7a None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:51.074759 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75030ed8-9501-4309-be12-ee35f93e300f None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:51.078905 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46704303-2e5c-4d0a-9b62-a79d19794f7a None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:51.079329 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02fcdd8-50f1-4d66-9be4-48301591916a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:51.079930 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02fcdd8-50f1-4d66-9be4-48301591916a admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:51.080210 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02fcdd8-50f1-4d66-9be4-48301591916a admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:51.080479 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02fcdd8-50f1-4d66-9be4-48301591916a admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:51.081115 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d02fcdd8-50f1-4d66-9be4-48301591916a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:51.082747 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d02fcdd8-50f1-4d66-9be4-48301591916a admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:51.084424 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a320725-14f4-4c82-9afa-d598c9c56748 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:51.085544 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a320725-14f4-4c82-9afa-d598c9c56748 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:51.085818 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a320725-14f4-4c82-9afa-d598c9c56748 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:51.086086 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a320725-14f4-4c82-9afa-d598c9c56748 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:51.086759 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a320725-14f4-4c82-9afa-d598c9c56748 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:51.088608 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a320725-14f4-4c82-9afa-d598c9c56748 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:51.095913 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 769/1563] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:44:51 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:44:51.097006 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551ce932-8346-4056-bb73-1acdad6f4a9b None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:51.097296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551ce932-8346-4056-bb73-1acdad6f4a9b None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:51.097574 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-551ce932-8346-4056-bb73-1acdad6f4a9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:51.121164 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 795/1564] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:44:51 2022] POST /identity/v3/projects => generated 385 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:44:51.121365 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-551ce932-8346-4056-bb73-1acdad6f4a9b None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:44:51.121468 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a320725-14f4-4c82-9afa-d598c9c56748 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:51.121560 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-551ce932-8346-4056-bb73-1acdad6f4a9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:44:51.180835 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 770/1565] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:44:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:44:51.181660 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-551ce932-8346-4056-bb73-1acdad6f4a9b None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:51.183118 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a320725-14f4-4c82-9afa-d598c9c56748 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:51.188514 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c5a114-78b3-4b19-b46d-4ca2330c15b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:51.189180 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c5a114-78b3-4b19-b46d-4ca2330c15b0 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:51.189413 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c5a114-78b3-4b19-b46d-4ca2330c15b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:51.189695 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c5a114-78b3-4b19-b46d-4ca2330c15b0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:51.190471 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c5a114-78b3-4b19-b46d-4ca2330c15b0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:44:51.190785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c5a114-78b3-4b19-b46d-4ca2330c15b0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:51.191313 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-551ce932-8346-4056-bb73-1acdad6f4a9b None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:51.192389 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c5a114-78b3-4b19-b46d-4ca2330c15b0 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:51.196414 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fea65cf-2da5-43fb-8193-029f0e11b280 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:51.197030 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fea65cf-2da5-43fb-8193-029f0e11b280 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:51.197308 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fea65cf-2da5-43fb-8193-029f0e11b280 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:51.197582 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fea65cf-2da5-43fb-8193-029f0e11b280 admin admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:51.198204 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fea65cf-2da5-43fb-8193-029f0e11b280 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:51.200143 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fea65cf-2da5-43fb-8193-029f0e11b280 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:51.212337 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 796/1566] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:44:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:44:51.212337 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c5a114-78b3-4b19-b46d-4ca2330c15b0 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:51.217491 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c5a114-78b3-4b19-b46d-4ca2330c15b0 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:51.223495 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da056b1-fc37-4700-8646-63eeda5b0610 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:51.224107 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da056b1-fc37-4700-8646-63eeda5b0610 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:51.224385 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da056b1-fc37-4700-8646-63eeda5b0610 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:51.224654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da056b1-fc37-4700-8646-63eeda5b0610 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:51.225311 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da056b1-fc37-4700-8646-63eeda5b0610 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:51.227275 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da056b1-fc37-4700-8646-63eeda5b0610 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:51.231103 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 771/1567] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:44:51 2022] POST /identity/v3/users => generated 289 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:44:51.231103 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fea65cf-2da5-43fb-8193-029f0e11b280 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:51.241562 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fea65cf-2da5-43fb-8193-029f0e11b280 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:51.247397 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730f11bf-85dc-405d-8dab-3bc87818fe63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:51.248010 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730f11bf-85dc-405d-8dab-3bc87818fe63 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:51.248292 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730f11bf-85dc-405d-8dab-3bc87818fe63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:51.248572 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730f11bf-85dc-405d-8dab-3bc87818fe63 admin admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:51.249218 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-730f11bf-85dc-405d-8dab-3bc87818fe63 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:44:51.249515 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-730f11bf-85dc-405d-8dab-3bc87818fe63 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:51.251158 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-730f11bf-85dc-405d-8dab-3bc87818fe63 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:51.251591 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 797/1568] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:44:51 2022] POST /identity/v3/users => generated 293 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:44:51.252421 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da056b1-fc37-4700-8646-63eeda5b0610 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:51.261997 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da056b1-fc37-4700-8646-63eeda5b0610 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:51.271605 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 772/1569] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:44:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:44:51.271783 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c828cd51-434c-414b-9e25-9a5d9242b4c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:51.271783 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c828cd51-434c-414b-9e25-9a5d9242b4c9 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:51.271972 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730f11bf-85dc-405d-8dab-3bc87818fe63 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:51.272617 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c828cd51-434c-414b-9e25-9a5d9242b4c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:51.272617 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c828cd51-434c-414b-9e25-9a5d9242b4c9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:51.272617 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c828cd51-434c-414b-9e25-9a5d9242b4c9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:44:51.272617 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c828cd51-434c-414b-9e25-9a5d9242b4c9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:51.272617 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c828cd51-434c-414b-9e25-9a5d9242b4c9 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:51.281992 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730f11bf-85dc-405d-8dab-3bc87818fe63 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:51.284584 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 798/1570] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:44:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:44:51.285208 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c828cd51-434c-414b-9e25-9a5d9242b4c9 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:51.294303 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d47b4e-7060-4e40-95f0-8352740cbe93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:51.294962 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d47b4e-7060-4e40-95f0-8352740cbe93 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:51.295244 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d47b4e-7060-4e40-95f0-8352740cbe93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:51.295514 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d47b4e-7060-4e40-95f0-8352740cbe93 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:51.296158 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d47b4e-7060-4e40-95f0-8352740cbe93 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:51.297767 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d47b4e-7060-4e40-95f0-8352740cbe93 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:51.297980 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c828cd51-434c-414b-9e25-9a5d9242b4c9 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:51.302569 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54c5c13-bc1a-46db-8b60-065c07415618 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:51.303245 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54c5c13-bc1a-46db-8b60-065c07415618 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:51.303551 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54c5c13-bc1a-46db-8b60-065c07415618 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:51.303826 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b54c5c13-bc1a-46db-8b60-065c07415618 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:51.304453 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b54c5c13-bc1a-46db-8b60-065c07415618 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:51.306323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b54c5c13-bc1a-46db-8b60-065c07415618 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:51.311284 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 773/1571] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:44:51 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:44:51.311284 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d47b4e-7060-4e40-95f0-8352740cbe93 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:51.319507 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d47b4e-7060-4e40-95f0-8352740cbe93 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:51.324555 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43632d94-787e-4d50-bdab-a805764fcce2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:51.325194 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43632d94-787e-4d50-bdab-a805764fcce2 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:51.325511 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43632d94-787e-4d50-bdab-a805764fcce2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:51.325789 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43632d94-787e-4d50-bdab-a805764fcce2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:51.326463 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43632d94-787e-4d50-bdab-a805764fcce2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:44:51.326772 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43632d94-787e-4d50-bdab-a805764fcce2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:51.328360 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43632d94-787e-4d50-bdab-a805764fcce2 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:51.330417 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 799/1572] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:44:51 2022] POST /identity/v3/users => generated 293 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:44:51.331470 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54c5c13-bc1a-46db-8b60-065c07415618 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:51.341638 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b54c5c13-bc1a-46db-8b60-065c07415618 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:51.343786 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 774/1573] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:44:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:44:51.344557 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43632d94-787e-4d50-bdab-a805764fcce2 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:51.353952 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75340bd-486a-4f1c-a9fd-22de321f0d35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:51.354626 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75340bd-486a-4f1c-a9fd-22de321f0d35 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:51.354764 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43632d94-787e-4d50-bdab-a805764fcce2 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:51.354909 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75340bd-486a-4f1c-a9fd-22de321f0d35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:51.355183 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75340bd-486a-4f1c-a9fd-22de321f0d35 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:51.355819 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d75340bd-486a-4f1c-a9fd-22de321f0d35 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:51.357840 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d75340bd-486a-4f1c-a9fd-22de321f0d35 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:51.359306 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a9abe4-7c64-4eb8-99fc-51ee0e49ab48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:51.359897 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a9abe4-7c64-4eb8-99fc-51ee0e49ab48 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:51.360175 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a9abe4-7c64-4eb8-99fc-51ee0e49ab48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:51.360446 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a9abe4-7c64-4eb8-99fc-51ee0e49ab48 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:51.361070 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71a9abe4-7c64-4eb8-99fc-51ee0e49ab48 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:51.362658 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71a9abe4-7c64-4eb8-99fc-51ee0e49ab48 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:51.372679 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 775/1574] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:44:51 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:44:51.373407 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a9abe4-7c64-4eb8-99fc-51ee0e49ab48 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:51.384331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a9abe4-7c64-4eb8-99fc-51ee0e49ab48 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:51.387324 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 800/1575] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:44:51 2022] POST /identity/v3/projects => generated 363 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:44:51.388181 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75340bd-486a-4f1c-a9fd-22de321f0d35 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:51.391516 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2aa7a07-c2b5-4d0d-ac32-509f80ce1a6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:51.392436 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2aa7a07-c2b5-4d0d-ac32-509f80ce1a6d admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:51.392724 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2aa7a07-c2b5-4d0d-ac32-509f80ce1a6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:51.393002 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2aa7a07-c2b5-4d0d-ac32-509f80ce1a6d admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:51.393642 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2aa7a07-c2b5-4d0d-ac32-509f80ce1a6d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:51.395360 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2aa7a07-c2b5-4d0d-ac32-509f80ce1a6d admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:51.406841 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 776/1576] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:44:51 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:44:51.407597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2aa7a07-c2b5-4d0d-ac32-509f80ce1a6d admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:51.417339 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2aa7a07-c2b5-4d0d-ac32-509f80ce1a6d admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:51.463568 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75340bd-486a-4f1c-a9fd-22de321f0d35 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:51.471808 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efff6709-e4dc-4786-bb44-947b259a269d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:51.472735 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efff6709-e4dc-4786-bb44-947b259a269d admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:51.473272 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efff6709-e4dc-4786-bb44-947b259a269d admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:51.473630 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:44:51.473750 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efff6709-e4dc-4786-bb44-947b259a269d admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:51.474477 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efff6709-e4dc-4786-bb44-947b259a269d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:51.477752 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efff6709-e4dc-4786-bb44-947b259a269d admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:51.480217 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e70c33a8-ea5a-49f5-90df-becf2b48523d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:51.480937 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70c33a8-ea5a-49f5-90df-becf2b48523d admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:51.481396 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70c33a8-ea5a-49f5-90df-becf2b48523d admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:51.481749 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70c33a8-ea5a-49f5-90df-becf2b48523d admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:51.482601 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e70c33a8-ea5a-49f5-90df-becf2b48523d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:51.484710 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e70c33a8-ea5a-49f5-90df-becf2b48523d admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:51.485058 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:44:51.514877 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 777/1578] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:44:51 2022] POST /identity/v3/projects => generated 375 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:44:51.515088 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 801/1578] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:44:51 2022] POST /identity/v3/projects => generated 387 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:44:51.515552 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efff6709-e4dc-4786-bb44-947b259a269d admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:51.518911 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e70c33a8-ea5a-49f5-90df-becf2b48523d admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:51.581514 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e70c33a8-ea5a-49f5-90df-becf2b48523d admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:51.581717 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efff6709-e4dc-4786-bb44-947b259a269d admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:51.586752 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6a4cd56-885c-4292-8106-b2c2ea2e87b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:51.587361 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a4cd56-885c-4292-8106-b2c2ea2e87b3 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:51.587644 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a4cd56-885c-4292-8106-b2c2ea2e87b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:51.587913 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a4cd56-885c-4292-8106-b2c2ea2e87b3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:51.588555 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6a4cd56-885c-4292-8106-b2c2ea2e87b3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:51.590154 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6a4cd56-885c-4292-8106-b2c2ea2e87b3 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:51.592783 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c39e5c8-75f4-49a1-96ab-aca6c52f0ce1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:51.593474 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c39e5c8-75f4-49a1-96ab-aca6c52f0ce1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:51.593763 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c39e5c8-75f4-49a1-96ab-aca6c52f0ce1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:51.594048 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c39e5c8-75f4-49a1-96ab-aca6c52f0ce1 admin admin] PATH_INFO: `/v3/projects/227ac6c4e642467ca3b9e0bf7181d5c9/users/ed9a4c24cbfb4689b7f984219c7503a4/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:51.603762 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 778/1579] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:44:51 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:44:51.603762 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6a4cd56-885c-4292-8106-b2c2ea2e87b3 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:51.609403 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c39e5c8-75f4-49a1-96ab-aca6c52f0ce1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=227ac6c4e642467ca3b9e0bf7181d5c9, user_id=ed9a4c24cbfb4689b7f984219c7503a4, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:51.610515 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: WARNING py.warnings [None req-1c39e5c8-75f4-49a1-96ab-aca6c52f0ce1 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 28 09:44:51.610515 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: warnings.warn(msg) Oct 28 09:44:51.610515 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: Oct 28 09:44:51.612455 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c39e5c8-75f4-49a1-96ab-aca6c52f0ce1 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:51.613328 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6a4cd56-885c-4292-8106-b2c2ea2e87b3 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:51.618168 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d07f0fd-847d-43b6-8064-b27f672c9354 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:51.618893 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d07f0fd-847d-43b6-8064-b27f672c9354 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:51.619149 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d07f0fd-847d-43b6-8064-b27f672c9354 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:51.619485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d07f0fd-847d-43b6-8064-b27f672c9354 admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:51.620176 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d07f0fd-847d-43b6-8064-b27f672c9354 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:51.622324 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d07f0fd-847d-43b6-8064-b27f672c9354 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:51.626792 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 802/1580] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:44:51 2022] PUT /identity/v3/projects/227ac6c4e642467ca3b9e0bf7181d5c9/users/ed9a4c24cbfb4689b7f984219c7503a4/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:44:51.627524 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c39e5c8-75f4-49a1-96ab-aca6c52f0ce1 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:51.654261 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 779/1581] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:44:51 2022] POST /identity/v3/projects => generated 403 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:44:51.655152 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d07f0fd-847d-43b6-8064-b27f672c9354 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:51.696152 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c39e5c8-75f4-49a1-96ab-aca6c52f0ce1 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:51.719144 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d07f0fd-847d-43b6-8064-b27f672c9354 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:51.723889 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1f1977-a8b7-4515-a19d-dc7dfca47459 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:51.724483 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1f1977-a8b7-4515-a19d-dc7dfca47459 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:51.724764 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1f1977-a8b7-4515-a19d-dc7dfca47459 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:51.725041 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1f1977-a8b7-4515-a19d-dc7dfca47459 admin admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:51.726254 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec1f1977-a8b7-4515-a19d-dc7dfca47459 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:51.728199 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec1f1977-a8b7-4515-a19d-dc7dfca47459 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:51.751719 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 780/1582] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:44:51 2022] POST /identity/v3/users => generated 282 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:44:51.752951 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1f1977-a8b7-4515-a19d-dc7dfca47459 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:51.756705 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-758131d3-ce83-4cb0-85d8-dc83c83e600e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:51.757371 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758131d3-ce83-4cb0-85d8-dc83c83e600e admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:51.757657 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758131d3-ce83-4cb0-85d8-dc83c83e600e admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:51.757963 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758131d3-ce83-4cb0-85d8-dc83c83e600e admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:51.758633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-758131d3-ce83-4cb0-85d8-dc83c83e600e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:51.760247 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-758131d3-ce83-4cb0-85d8-dc83c83e600e admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:51.763184 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1f1977-a8b7-4515-a19d-dc7dfca47459 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:51.767860 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67efdee4-9a8c-4bd2-b6c5-1c04652cce5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:51.768453 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67efdee4-9a8c-4bd2-b6c5-1c04652cce5d admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:51.768733 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67efdee4-9a8c-4bd2-b6c5-1c04652cce5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:51.769010 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67efdee4-9a8c-4bd2-b6c5-1c04652cce5d admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:51.769647 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67efdee4-9a8c-4bd2-b6c5-1c04652cce5d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:51.773861 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 803/1583] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:44:51 2022] GET /identity/v3/roles => generated 1420 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:44:51.773861 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-758131d3-ce83-4cb0-85d8-dc83c83e600e admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:51.774089 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67efdee4-9a8c-4bd2-b6c5-1c04652cce5d admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:51.781433 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-758131d3-ce83-4cb0-85d8-dc83c83e600e admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:51.782203 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 781/1584] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:44:51 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:44:51.783166 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67efdee4-9a8c-4bd2-b6c5-1c04652cce5d admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:51.787012 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b97b46-5287-4e08-9629-43ae8a72b083 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:51.787647 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b97b46-5287-4e08-9629-43ae8a72b083 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:51.787981 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b97b46-5287-4e08-9629-43ae8a72b083 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:51.789004 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b97b46-5287-4e08-9629-43ae8a72b083 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:51.789609 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b97b46-5287-4e08-9629-43ae8a72b083 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:51.792389 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b97b46-5287-4e08-9629-43ae8a72b083 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:51.793751 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67efdee4-9a8c-4bd2-b6c5-1c04652cce5d admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:51.798573 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae6ba6b4-f719-4237-a728-ae4ac8157730 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:51.799168 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6ba6b4-f719-4237-a728-ae4ac8157730 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:51.799452 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6ba6b4-f719-4237-a728-ae4ac8157730 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:51.799725 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6ba6b4-f719-4237-a728-ae4ac8157730 admin admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:51.800353 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae6ba6b4-f719-4237-a728-ae4ac8157730 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:51.802247 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae6ba6b4-f719-4237-a728-ae4ac8157730 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:51.803768 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:44:51.814562 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 804/1585] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:44:51 2022] POST /identity/v3/users => generated 294 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:44:51.815457 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b97b46-5287-4e08-9629-43ae8a72b083 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:51.825628 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b97b46-5287-4e08-9629-43ae8a72b083 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:51.831467 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 782/1586] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:44:51 2022] POST /identity/v3/users => generated 288 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:44:51.831467 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fb896a-8cf0-4aa2-af1e-cc97ad6653c2 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:51.831467 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fb896a-8cf0-4aa2-af1e-cc97ad6653c2 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:51.831467 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fb896a-8cf0-4aa2-af1e-cc97ad6653c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:51.832751 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a95dae-6657-4567-9eff-719daf5b7134 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:51.833428 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a95dae-6657-4567-9eff-719daf5b7134 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:51.833781 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a95dae-6657-4567-9eff-719daf5b7134 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:51.834134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a95dae-6657-4567-9eff-719daf5b7134 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:51.835386 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60a95dae-6657-4567-9eff-719daf5b7134 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:51.836959 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60a95dae-6657-4567-9eff-719daf5b7134 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:51.847781 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 805/1587] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:44:51 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:44:51.848595 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a95dae-6657-4567-9eff-719daf5b7134 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:51.849425 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-e1fb896a-8cf0-4aa2-af1e-cc97ad6653c2 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:44:51.851242 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-e1fb896a-8cf0-4aa2-af1e-cc97ad6653c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:44:51.858811 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a95dae-6657-4567-9eff-719daf5b7134 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:51.858976 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 783/1588] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:44:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:44:51.859670 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1fb896a-8cf0-4aa2-af1e-cc97ad6653c2 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:51.866072 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee561b6-cbc7-4ca3-bf76-ac62f44dbcd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:51.866669 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee561b6-cbc7-4ca3-bf76-ac62f44dbcd9 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:51.867056 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee561b6-cbc7-4ca3-bf76-ac62f44dbcd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:51.867226 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee561b6-cbc7-4ca3-bf76-ac62f44dbcd9 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:51.867841 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ee561b6-cbc7-4ca3-bf76-ac62f44dbcd9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:51.869644 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1fb896a-8cf0-4aa2-af1e-cc97ad6653c2 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:51.870041 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ee561b6-cbc7-4ca3-bf76-ac62f44dbcd9 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:51.874238 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce1d9a49-57b4-43c8-b25b-b4fea2eb4d68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:51.874874 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1d9a49-57b4-43c8-b25b-b4fea2eb4d68 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:51.875159 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1d9a49-57b4-43c8-b25b-b4fea2eb4d68 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:51.875433 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1d9a49-57b4-43c8-b25b-b4fea2eb4d68 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:51.876107 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce1d9a49-57b4-43c8-b25b-b4fea2eb4d68 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:51.877759 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce1d9a49-57b4-43c8-b25b-b4fea2eb4d68 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:51.884501 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:44:51.890690 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 784/1589] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:44:51 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:44:51.891778 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce1d9a49-57b4-43c8-b25b-b4fea2eb4d68 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:51.893599 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 806/1590] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:44:51 2022] POST /identity/v3/users => generated 302 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:44:51.894693 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee561b6-cbc7-4ca3-bf76-ac62f44dbcd9 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:51.905576 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee561b6-cbc7-4ca3-bf76-ac62f44dbcd9 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:51.907266 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce1d9a49-57b4-43c8-b25b-b4fea2eb4d68 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:51.910331 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f561e946-b98e-4b09-99c0-98cc203c37d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:51.911032 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f561e946-b98e-4b09-99c0-98cc203c37d6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:51.911315 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f561e946-b98e-4b09-99c0-98cc203c37d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:51.911595 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f561e946-b98e-4b09-99c0-98cc203c37d6 admin admin] PATH_INFO: `/v3/projects/1d65871da7a445fda2f75ea82af5f9b9/users/f63cfb4896ff45848bbd79a65ef0cdf1/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:51.911896 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a35f03-43d7-48d6-95a8-94b6877c73f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:51.912543 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a35f03-43d7-48d6-95a8-94b6877c73f6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:51.913379 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a35f03-43d7-48d6-95a8-94b6877c73f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:51.913661 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a35f03-43d7-48d6-95a8-94b6877c73f6 admin admin] PATH_INFO: `/v3/projects/41516f10d050497e82056ddd1be5981c/users/e451d264e4964409b09e0b96ae8b983e/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:51.927551 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f561e946-b98e-4b09-99c0-98cc203c37d6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1d65871da7a445fda2f75ea82af5f9b9, user_id=f63cfb4896ff45848bbd79a65ef0cdf1, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:51.929060 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a35f03-43d7-48d6-95a8-94b6877c73f6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=41516f10d050497e82056ddd1be5981c, user_id=e451d264e4964409b09e0b96ae8b983e, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:51.930246 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f561e946-b98e-4b09-99c0-98cc203c37d6 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:51.931743 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a35f03-43d7-48d6-95a8-94b6877c73f6 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:51.944236 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 807/1591] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:44:51 2022] PUT /identity/v3/projects/1d65871da7a445fda2f75ea82af5f9b9/users/f63cfb4896ff45848bbd79a65ef0cdf1/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:44:51.945054 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f561e946-b98e-4b09-99c0-98cc203c37d6 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:51.949082 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 785/1592] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:44:51 2022] PUT /identity/v3/projects/41516f10d050497e82056ddd1be5981c/users/e451d264e4964409b09e0b96ae8b983e/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:44:51.949954 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a35f03-43d7-48d6-95a8-94b6877c73f6 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:52.016624 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f561e946-b98e-4b09-99c0-98cc203c37d6 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:52.018009 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a35f03-43d7-48d6-95a8-94b6877c73f6 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:52.021537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b6400fb-17d2-4dff-bf61-a94e559576fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:52.022714 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6400fb-17d2-4dff-bf61-a94e559576fc admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:52.022997 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6400fb-17d2-4dff-bf61-a94e559576fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:52.023270 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6400fb-17d2-4dff-bf61-a94e559576fc admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:52.023446 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77e2c4b7-4d91-422a-aeb5-4fd5e047bc8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:52.023909 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b6400fb-17d2-4dff-bf61-a94e559576fc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:52.024067 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e2c4b7-4d91-422a-aeb5-4fd5e047bc8e admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:52.024349 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e2c4b7-4d91-422a-aeb5-4fd5e047bc8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:52.024620 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77e2c4b7-4d91-422a-aeb5-4fd5e047bc8e admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:52.025275 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77e2c4b7-4d91-422a-aeb5-4fd5e047bc8e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:52.025628 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b6400fb-17d2-4dff-bf61-a94e559576fc admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:52.027037 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77e2c4b7-4d91-422a-aeb5-4fd5e047bc8e admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:52.036037 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 808/1593] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:44:51 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:44:52.036978 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b6400fb-17d2-4dff-bf61-a94e559576fc admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:52.042797 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 786/1594] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:44:51 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:44:52.042797 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77e2c4b7-4d91-422a-aeb5-4fd5e047bc8e admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:52.044634 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-77e2c4b7-4d91-422a-aeb5-4fd5e047bc8e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:44:52.047148 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b6400fb-17d2-4dff-bf61-a94e559576fc admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:52.051748 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e96f501-5d7a-4773-9b0b-6bdc3542eefe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:52.052387 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e96f501-5d7a-4773-9b0b-6bdc3542eefe admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:52.052673 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e96f501-5d7a-4773-9b0b-6bdc3542eefe admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:52.052950 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e96f501-5d7a-4773-9b0b-6bdc3542eefe admin admin] PATH_INFO: `/v3/projects/4f351d016ace4e369be852bb7aa916fe/users/e65d298cca0f478eba502edf5bd13954/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:52.069702 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e96f501-5d7a-4773-9b0b-6bdc3542eefe admin admin] RBAC: Authorizing `identity:create_grant(project_id=4f351d016ace4e369be852bb7aa916fe, user_id=e65d298cca0f478eba502edf5bd13954, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:52.072374 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e96f501-5d7a-4773-9b0b-6bdc3542eefe admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:52.085613 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 809/1595] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:44:52 2022] PUT /identity/v3/projects/4f351d016ace4e369be852bb7aa916fe/users/e65d298cca0f478eba502edf5bd13954/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:44:52.086536 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e96f501-5d7a-4773-9b0b-6bdc3542eefe admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:52.107396 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77e2c4b7-4d91-422a-aeb5-4fd5e047bc8e admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:52.148778 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e96f501-5d7a-4773-9b0b-6bdc3542eefe admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:52.153446 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b2675c-f1d8-443f-891e-d54f0270618f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:52.154081 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b2675c-f1d8-443f-891e-d54f0270618f admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:52.154367 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b2675c-f1d8-443f-891e-d54f0270618f admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:52.154675 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90b2675c-f1d8-443f-891e-d54f0270618f admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:52.155317 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b2675c-f1d8-443f-891e-d54f0270618f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:52.155908 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db05f49-d73e-4c4b-be88-33a7b0b51b54 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:52.156554 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db05f49-d73e-4c4b-be88-33a7b0b51b54 service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:52.156837 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db05f49-d73e-4c4b-be88-33a7b0b51b54 service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:52.156957 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90b2675c-f1d8-443f-891e-d54f0270618f admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:52.157116 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db05f49-d73e-4c4b-be88-33a7b0b51b54 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:52.167062 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8db05f49-d73e-4c4b-be88-33a7b0b51b54 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:52.167786 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 810/1596] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:44:52 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:44:52.168502 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8db05f49-d73e-4c4b-be88-33a7b0b51b54 service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:52.168636 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b2675c-f1d8-443f-891e-d54f0270618f admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:52.178659 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 787/1597] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:44:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:44:52.179904 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbfcd5a1-8ff6-4aec-bde3-b95292c22cb1 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:52.180195 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbfcd5a1-8ff6-4aec-bde3-b95292c22cb1 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:52.180475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbfcd5a1-8ff6-4aec-bde3-b95292c22cb1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:52.182386 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90b2675c-f1d8-443f-891e-d54f0270618f admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:52.190917 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a57cb6e-6526-4700-9a54-91df29812439 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:52.191930 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a57cb6e-6526-4700-9a54-91df29812439 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:52.192465 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a57cb6e-6526-4700-9a54-91df29812439 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:52.192924 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a57cb6e-6526-4700-9a54-91df29812439 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:52.194566 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a57cb6e-6526-4700-9a54-91df29812439 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:52.197560 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a57cb6e-6526-4700-9a54-91df29812439 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:52.201462 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-bbfcd5a1-8ff6-4aec-bde3-b95292c22cb1 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:44:52.202789 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-bbfcd5a1-8ff6-4aec-bde3-b95292c22cb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:44:52.207581 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:44:52.214535 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 788/1598] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:44:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:44:52.216918 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd69b00-627b-4796-8228-c79f8352e39b None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:52.217100 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd69b00-627b-4796-8228-c79f8352e39b None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:52.217539 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd69b00-627b-4796-8228-c79f8352e39b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:52.221808 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 811/1599] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:44:52 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:44:52.221808 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a57cb6e-6526-4700-9a54-91df29812439 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:52.245060 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a57cb6e-6526-4700-9a54-91df29812439 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:52.254909 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ff8550-bac9-4a0c-a9f8-fb77a6a3c4db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:52.255952 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ff8550-bac9-4a0c-a9f8-fb77a6a3c4db admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:52.256494 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ff8550-bac9-4a0c-a9f8-fb77a6a3c4db admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:52.256988 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ff8550-bac9-4a0c-a9f8-fb77a6a3c4db admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:52.258782 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51ff8550-bac9-4a0c-a9f8-fb77a6a3c4db admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:52.261841 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51ff8550-bac9-4a0c-a9f8-fb77a6a3c4db admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:52.272708 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-edd69b00-627b-4796-8228-c79f8352e39b None None] MFA Rules not processed for user `f63cfb4896ff45848bbd79a65ef0cdf1`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:44:52.274155 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-edd69b00-627b-4796-8228-c79f8352e39b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:44:52.285095 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 812/1600] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:44:52 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:44:52.285095 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ff8550-bac9-4a0c-a9f8-fb77a6a3c4db admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:52.293932 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:44:52.302241 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ff8550-bac9-4a0c-a9f8-fb77a6a3c4db admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:52.310706 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e666c8e7-b312-48a6-b961-3725ec523a4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:52.312136 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e666c8e7-b312-48a6-b961-3725ec523a4d admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:52.312405 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e666c8e7-b312-48a6-b961-3725ec523a4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:52.312983 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e666c8e7-b312-48a6-b961-3725ec523a4d admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:52.314188 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e666c8e7-b312-48a6-b961-3725ec523a4d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:52.317158 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e666c8e7-b312-48a6-b961-3725ec523a4d admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:52.334965 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 813/1601] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:44:52 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:44:52.337143 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537bb71e-be9b-48e9-a82f-cb28ad9a9ceb None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:52.337595 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537bb71e-be9b-48e9-a82f-cb28ad9a9ceb None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:52.338243 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537bb71e-be9b-48e9-a82f-cb28ad9a9ceb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:52.361785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-537bb71e-be9b-48e9-a82f-cb28ad9a9ceb None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:44:52.363121 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-537bb71e-be9b-48e9-a82f-cb28ad9a9ceb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:44:52.374656 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 814/1602] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:44:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:44:52.374656 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-537bb71e-be9b-48e9-a82f-cb28ad9a9ceb None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:52.388436 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 789/1603] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:44:52 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:44:52.388436 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd69b00-627b-4796-8228-c79f8352e39b None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:52.395686 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-537bb71e-be9b-48e9-a82f-cb28ad9a9ceb None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:52.401852 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14a700f-1d53-4bde-9e9e-3faa862705c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:52.402847 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14a700f-1d53-4bde-9e9e-3faa862705c6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:52.402985 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14a700f-1d53-4bde-9e9e-3faa862705c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:52.403285 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14a700f-1d53-4bde-9e9e-3faa862705c6 admin admin] PATH_INFO: `/v3/projects/385349b18e6f4217adbcec1fb43b1e6b/users/d9e9bb6b29814e4d8a3e7bdeec4bec58/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:52.404082 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd69b00-627b-4796-8228-c79f8352e39b None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:52.410456 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f3c535-a189-4804-a49c-28ab8a9bea91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:52.411652 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f3c535-a189-4804-a49c-28ab8a9bea91 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:52.411652 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f3c535-a189-4804-a49c-28ab8a9bea91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:52.411652 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f3c535-a189-4804-a49c-28ab8a9bea91 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:52.412243 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19f3c535-a189-4804-a49c-28ab8a9bea91 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:52.417082 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19f3c535-a189-4804-a49c-28ab8a9bea91 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:52.427242 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a14a700f-1d53-4bde-9e9e-3faa862705c6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=385349b18e6f4217adbcec1fb43b1e6b, user_id=d9e9bb6b29814e4d8a3e7bdeec4bec58, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:52.430818 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a14a700f-1d53-4bde-9e9e-3faa862705c6 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:52.435840 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 790/1604] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:44:52 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:44:52.435840 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f3c535-a189-4804-a49c-28ab8a9bea91 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:52.447771 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f3c535-a189-4804-a49c-28ab8a9bea91 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:52.462650 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 815/1605] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:44:52 2022] PUT /identity/v3/projects/385349b18e6f4217adbcec1fb43b1e6b/users/d9e9bb6b29814e4d8a3e7bdeec4bec58/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 82 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:44:52.462650 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14a700f-1d53-4bde-9e9e-3faa862705c6 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:52.508732 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae31ab3-7cba-4876-85cf-ab2380ed8ec2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:52.509760 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae31ab3-7cba-4876-85cf-ab2380ed8ec2 service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:52.510072 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae31ab3-7cba-4876-85cf-ab2380ed8ec2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:52.510350 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae31ab3-7cba-4876-85cf-ab2380ed8ec2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:52.532733 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14a700f-1d53-4bde-9e9e-3faa862705c6 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:52.538707 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5bd234d-691b-4a7a-92fb-1cd3ab4ad89f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:52.539640 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5bd234d-691b-4a7a-92fb-1cd3ab4ad89f admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:52.540127 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5bd234d-691b-4a7a-92fb-1cd3ab4ad89f admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:52.540463 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5bd234d-691b-4a7a-92fb-1cd3ab4ad89f admin admin] PATH_INFO: `/v3/projects/29c47b13a2044939b5446ffe5397633a/users/715a782e5a56431e88e4fe5a3b0a62aa/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:52.558729 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5bd234d-691b-4a7a-92fb-1cd3ab4ad89f admin admin] RBAC: Authorizing `identity:create_grant(project_id=29c47b13a2044939b5446ffe5397633a, user_id=715a782e5a56431e88e4fe5a3b0a62aa, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:52.561649 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5bd234d-691b-4a7a-92fb-1cd3ab4ad89f admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:52.584342 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 816/1606] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:44:52 2022] PUT /identity/v3/projects/29c47b13a2044939b5446ffe5397633a/users/715a782e5a56431e88e4fe5a3b0a62aa/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:44:52.585716 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5bd234d-691b-4a7a-92fb-1cd3ab4ad89f admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:52.590387 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae31ab3-7cba-4876-85cf-ab2380ed8ec2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:52.592691 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae31ab3-7cba-4876-85cf-ab2380ed8ec2 service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:52.607727 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 791/1607] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:44:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:44:52.608993 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae31ab3-7cba-4876-85cf-ab2380ed8ec2 service neutron] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:52.618693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae31ab3-7cba-4876-85cf-ab2380ed8ec2 service neutron] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:52.667340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5bd234d-691b-4a7a-92fb-1cd3ab4ad89f admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:52.672352 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b29bd3-f219-44c3-a5b9-6b8ac43f1970 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:52.673114 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b29bd3-f219-44c3-a5b9-6b8ac43f1970 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:52.673367 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b29bd3-f219-44c3-a5b9-6b8ac43f1970 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:52.673809 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b29bd3-f219-44c3-a5b9-6b8ac43f1970 admin admin] PATH_INFO: `/v3/projects/f2fcce993e6b4388bde493f0702fd49d/users/a4237ca76acd4f7483aaf66e1f5d2eea/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:52.698641 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed8f5f12-dde5-4c7c-9c00-9162a07a49a5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:52.699270 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8f5f12-dde5-4c7c-9c00-9162a07a49a5 service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:52.699548 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8f5f12-dde5-4c7c-9c00-9162a07a49a5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:52.699873 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8f5f12-dde5-4c7c-9c00-9162a07a49a5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:52.703353 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51b29bd3-f219-44c3-a5b9-6b8ac43f1970 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f2fcce993e6b4388bde493f0702fd49d, user_id=a4237ca76acd4f7483aaf66e1f5d2eea, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:52.707756 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51b29bd3-f219-44c3-a5b9-6b8ac43f1970 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:52.714859 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:44:52.717744 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed8f5f12-dde5-4c7c-9c00-9162a07a49a5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:52.720453 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed8f5f12-dde5-4c7c-9c00-9162a07a49a5 service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:52.731459 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 817/1608] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:44:52 2022] PUT /identity/v3/projects/f2fcce993e6b4388bde493f0702fd49d/users/a4237ca76acd4f7483aaf66e1f5d2eea/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:44:52.733644 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740f0af1-2146-47ec-ab05-cb05b672eb1f None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:52.733929 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740f0af1-2146-47ec-ab05-cb05b672eb1f None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:52.734205 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740f0af1-2146-47ec-ab05-cb05b672eb1f None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:52.744813 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 818/1609] 10.222.0.25 () {54 vars in 960 bytes} [Fri Oct 28 09:44:52 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:44:52.746745 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740f0af1-2146-47ec-ab05-cb05b672eb1f None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:52.750594 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 792/1610] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:44:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:44:52.752289 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31d3675-fcec-498e-9db8-9c015471520c None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:52.752711 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31d3675-fcec-498e-9db8-9c015471520c None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:52.753135 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31d3675-fcec-498e-9db8-9c015471520c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:52.770285 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:44:52.787365 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-a31d3675-fcec-498e-9db8-9c015471520c None None] MFA Rules not processed for user `d9e9bb6b29814e4d8a3e7bdeec4bec58`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:44:52.790911 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-a31d3675-fcec-498e-9db8-9c015471520c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:44:52.845557 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740f0af1-2146-47ec-ab05-cb05b672eb1f None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:52.850781 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5cf655-cea8-46d6-932e-b87eaf46b277 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:52.851481 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5cf655-cea8-46d6-932e-b87eaf46b277 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:52.851925 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5cf655-cea8-46d6-932e-b87eaf46b277 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:52.852204 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5cf655-cea8-46d6-932e-b87eaf46b277 admin admin] PATH_INFO: `/v3/projects/123dc2248dae40edbaa0d87ceabfbe1e/users/92fd6faa43c44bfba111514ea3453e7d/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:52.869237 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d5cf655-cea8-46d6-932e-b87eaf46b277 admin admin] RBAC: Authorizing `identity:create_grant(project_id=123dc2248dae40edbaa0d87ceabfbe1e, user_id=92fd6faa43c44bfba111514ea3453e7d, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:52.872118 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d5cf655-cea8-46d6-932e-b87eaf46b277 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:52.873275 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 793/1611] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:44:52 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:44:52.874343 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8da77d9-5a59-49e4-8624-87327d3bbbae None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:52.874664 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8da77d9-5a59-49e4-8624-87327d3bbbae None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:52.874991 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8da77d9-5a59-49e4-8624-87327d3bbbae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:52.891653 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 819/1612] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:44:52 2022] PUT /identity/v3/projects/123dc2248dae40edbaa0d87ceabfbe1e/users/92fd6faa43c44bfba111514ea3453e7d/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:44:52.893018 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c12f65-e196-4c95-9ac3-5bab4e569919 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:52.893312 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c12f65-e196-4c95-9ac3-5bab4e569919 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:52.893984 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c12f65-e196-4c95-9ac3-5bab4e569919 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:52.905478 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-e8da77d9-5a59-49e4-8624-87327d3bbbae None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:44:52.906741 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-e8da77d9-5a59-49e4-8624-87327d3bbbae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:44:52.919819 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-20c12f65-e196-4c95-9ac3-5bab4e569919 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:44:52.921206 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-20c12f65-e196-4c95-9ac3-5bab4e569919 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:44:52.987322 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 794/1613] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:44:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:44:52.987322 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af035eb-29fb-4d83-8850-da394d2f193d None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:52.987322 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af035eb-29fb-4d83-8850-da394d2f193d None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:52.987322 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af035eb-29fb-4d83-8850-da394d2f193d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:53.014352 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 820/1614] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:44:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:44:53.014352 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c12f65-e196-4c95-9ac3-5bab4e569919 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:53.026245 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c12f65-e196-4c95-9ac3-5bab4e569919 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:53.033009 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1090470-2856-46d4-936e-61955a424271 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:53.033606 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1090470-2856-46d4-936e-61955a424271 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:53.033892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1090470-2856-46d4-936e-61955a424271 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:53.034326 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1090470-2856-46d4-936e-61955a424271 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:53.034942 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1090470-2856-46d4-936e-61955a424271 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:53.037290 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1090470-2856-46d4-936e-61955a424271 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:53.039592 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-6af035eb-29fb-4d83-8850-da394d2f193d None None] MFA Rules not processed for user `688ceb201d304c2da5373c3cf36c2d79`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:44:53.041176 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-6af035eb-29fb-4d83-8850-da394d2f193d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:44:53.073316 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 821/1615] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:44:53 2022] POST /identity/v3/projects => generated 365 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:44:53.074800 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9319352-cdc5-4dd2-a33b-00fb556dea25 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:53.075099 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9319352-cdc5-4dd2-a33b-00fb556dea25 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:53.075379 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9319352-cdc5-4dd2-a33b-00fb556dea25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:53.107741 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-f9319352-cdc5-4dd2-a33b-00fb556dea25 None None] MFA Rules not processed for user `92fd6faa43c44bfba111514ea3453e7d`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:44:53.109111 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-f9319352-cdc5-4dd2-a33b-00fb556dea25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:44:53.125462 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 795/1616] 10.222.0.25 () {58 vars in 1057 bytes} [Fri Oct 28 09:44:52 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:44:53.126206 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af035eb-29fb-4d83-8850-da394d2f193d None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:53.134423 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:44:53.196641 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 822/1617] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:44:53 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:44:53.197620 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9319352-cdc5-4dd2-a33b-00fb556dea25 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:53.201669 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af035eb-29fb-4d83-8850-da394d2f193d None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:53.204619 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:44:53.267156 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b2c3e6-d55c-488d-a947-28befc5226bd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:53.267784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b2c3e6-d55c-488d-a947-28befc5226bd service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:53.268068 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b2c3e6-d55c-488d-a947-28befc5226bd service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:53.268344 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b2c3e6-d55c-488d-a947-28befc5226bd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:53.282495 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9319352-cdc5-4dd2-a33b-00fb556dea25 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:53.286938 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67009b59-fbaf-4b42-a3dd-d2b17c96ed7b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:53.288661 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67009b59-fbaf-4b42-a3dd-d2b17c96ed7b service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:53.288661 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67009b59-fbaf-4b42-a3dd-d2b17c96ed7b service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:53.288939 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67009b59-fbaf-4b42-a3dd-d2b17c96ed7b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:53.341574 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6b2c3e6-d55c-488d-a947-28befc5226bd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:53.343616 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6b2c3e6-d55c-488d-a947-28befc5226bd service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:53.354434 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 796/1618] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:44:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 229 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:44:53.355591 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b2c3e6-d55c-488d-a947-28befc5226bd service neutron] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:53.362701 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67009b59-fbaf-4b42-a3dd-d2b17c96ed7b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:53.364150 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67009b59-fbaf-4b42-a3dd-d2b17c96ed7b service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:53.367167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b2c3e6-d55c-488d-a947-28befc5226bd service neutron] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:53.373418 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae174af-a4e5-4c24-bbd0-8b3c4930e0ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:53.374202 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae174af-a4e5-4c24-bbd0-8b3c4930e0ee admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:53.374489 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae174af-a4e5-4c24-bbd0-8b3c4930e0ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:53.374798 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae174af-a4e5-4c24-bbd0-8b3c4930e0ee admin admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:53.375447 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ae174af-a4e5-4c24-bbd0-8b3c4930e0ee admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:53.376593 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 823/1619] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:44:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:44:53.377454 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67009b59-fbaf-4b42-a3dd-d2b17c96ed7b service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:53.377570 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ae174af-a4e5-4c24-bbd0-8b3c4930e0ee admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:53.412121 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 797/1620] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:44:53 2022] POST /identity/v3/users => generated 289 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:44:53.413273 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae174af-a4e5-4c24-bbd0-8b3c4930e0ee admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:53.438628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae174af-a4e5-4c24-bbd0-8b3c4930e0ee admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:53.447086 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7922fb-bf5c-4077-9030-d3aa1839e009 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:53.448065 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7922fb-bf5c-4077-9030-d3aa1839e009 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:53.448272 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7922fb-bf5c-4077-9030-d3aa1839e009 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:53.448784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7922fb-bf5c-4077-9030-d3aa1839e009 admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:53.449185 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b7922fb-bf5c-4077-9030-d3aa1839e009 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:53.451384 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b7922fb-bf5c-4077-9030-d3aa1839e009 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:53.509743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67009b59-fbaf-4b42-a3dd-d2b17c96ed7b service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:53.517939 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 798/1621] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:44:53 2022] POST /identity/v3/projects => generated 405 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:44:53.519522 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7922fb-bf5c-4077-9030-d3aa1839e009 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:53.525544 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05125b6b-a630-42cb-8f00-54c74e4f2aed service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:53.526130 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05125b6b-a630-42cb-8f00-54c74e4f2aed service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:53.526425 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05125b6b-a630-42cb-8f00-54c74e4f2aed service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:53.527209 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05125b6b-a630-42cb-8f00-54c74e4f2aed service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:53.607108 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:44:53.634428 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:44:53.653617 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7922fb-bf5c-4077-9030-d3aa1839e009 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:53.655591 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05125b6b-a630-42cb-8f00-54c74e4f2aed service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:53.659121 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05125b6b-a630-42cb-8f00-54c74e4f2aed service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:53.664701 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2cef2f-42ec-4ab3-ae35-433c7c119336 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:53.666602 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2cef2f-42ec-4ab3-ae35-433c7c119336 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:53.668064 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2cef2f-42ec-4ab3-ae35-433c7c119336 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:53.668975 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2cef2f-42ec-4ab3-ae35-433c7c119336 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:53.670432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b2cef2f-42ec-4ab3-ae35-433c7c119336 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:53.671945 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 824/1622] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:44:53 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 295 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:44:53.679271 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b2cef2f-42ec-4ab3-ae35-433c7c119336 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:53.685378 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05125b6b-a630-42cb-8f00-54c74e4f2aed service nova] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:53.707296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 799/1623] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:44:53 2022] GET /identity/v3/roles => generated 1420 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:44:53.712260 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2cef2f-42ec-4ab3-ae35-433c7c119336 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:53.715698 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05125b6b-a630-42cb-8f00-54c74e4f2aed service nova] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:53.733057 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7663546-9995-4f8b-b06f-84c0083eb5a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:53.733365 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7663546-9995-4f8b-b06f-84c0083eb5a4 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:53.733655 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7663546-9995-4f8b-b06f-84c0083eb5a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:53.733930 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7663546-9995-4f8b-b06f-84c0083eb5a4 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:53.738078 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7663546-9995-4f8b-b06f-84c0083eb5a4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:53.740106 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7663546-9995-4f8b-b06f-84c0083eb5a4 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:53.747115 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2cef2f-42ec-4ab3-ae35-433c7c119336 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:53.760938 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b7cbdc-77f7-403a-867e-bc4f4fb7ef00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:53.762139 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b7cbdc-77f7-403a-867e-bc4f4fb7ef00 admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:53.762736 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b7cbdc-77f7-403a-867e-bc4f4fb7ef00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:53.763313 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b7cbdc-77f7-403a-867e-bc4f4fb7ef00 admin admin] PATH_INFO: `/v3/projects/0f3ccd6d880e4fa19e439b73f903870f/users/20f9a30e0c0e422c8eedb98d46018fc9/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:53.794329 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 825/1624] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:44:53 2022] POST /identity/v3/users => generated 309 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:44:53.795267 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4b7cbdc-77f7-403a-867e-bc4f4fb7ef00 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0f3ccd6d880e4fa19e439b73f903870f, user_id=20f9a30e0c0e422c8eedb98d46018fc9, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:53.795385 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7663546-9995-4f8b-b06f-84c0083eb5a4 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:53.800488 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4b7cbdc-77f7-403a-867e-bc4f4fb7ef00 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:53.817112 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7663546-9995-4f8b-b06f-84c0083eb5a4 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:53.817999 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 800/1625] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:44:53 2022] PUT /identity/v3/projects/0f3ccd6d880e4fa19e439b73f903870f/users/20f9a30e0c0e422c8eedb98d46018fc9/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:44:53.823195 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b7cbdc-77f7-403a-867e-bc4f4fb7ef00 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:53.912207 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b7cbdc-77f7-403a-867e-bc4f4fb7ef00 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:53.921411 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84fbd683-443b-43ec-9019-9fbe2321a8fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:53.923420 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84fbd683-443b-43ec-9019-9fbe2321a8fc admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:53.923993 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84fbd683-443b-43ec-9019-9fbe2321a8fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:53.924558 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84fbd683-443b-43ec-9019-9fbe2321a8fc admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:53.926417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84fbd683-443b-43ec-9019-9fbe2321a8fc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:53.929645 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84fbd683-443b-43ec-9019-9fbe2321a8fc admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:53.944457 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 801/1626] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:44:53 2022] GET /identity/v3/roles => generated 1420 bytes in 122 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:44:53.948902 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84fbd683-443b-43ec-9019-9fbe2321a8fc admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:53.960981 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7f0abd-5f22-45fe-b6b8-db765dca6086 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:53.961639 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7f0abd-5f22-45fe-b6b8-db765dca6086 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:53.962102 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7f0abd-5f22-45fe-b6b8-db765dca6086 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:53.962436 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7f0abd-5f22-45fe-b6b8-db765dca6086 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:53.963143 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84fbd683-443b-43ec-9019-9fbe2321a8fc admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:53.963290 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd7f0abd-5f22-45fe-b6b8-db765dca6086 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:53.965097 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd7f0abd-5f22-45fe-b6b8-db765dca6086 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:53.971485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172182a4-42fd-4c93-b049-8f879990a6ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:53.974207 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172182a4-42fd-4c93-b049-8f879990a6ad admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:53.974958 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172182a4-42fd-4c93-b049-8f879990a6ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:53.975767 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172182a4-42fd-4c93-b049-8f879990a6ad admin admin] PATH_INFO: `/v3/projects/0f3ccd6d880e4fa19e439b73f903870f/users/20f9a30e0c0e422c8eedb98d46018fc9/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:53.979524 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172182a4-42fd-4c93-b049-8f879990a6ad admin admin] RBAC: Authorizing `identity:create_grant(project_id=0f3ccd6d880e4fa19e439b73f903870f, user_id=20f9a30e0c0e422c8eedb98d46018fc9, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:53.980175 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 826/1627] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:44:53 2022] GET /identity/v3/roles => generated 1420 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:44:53.982793 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172182a4-42fd-4c93-b049-8f879990a6ad admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:53.986462 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7f0abd-5f22-45fe-b6b8-db765dca6086 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:54.001756 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7f0abd-5f22-45fe-b6b8-db765dca6086 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:54.002816 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 802/1628] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:44:53 2022] PUT /identity/v3/projects/0f3ccd6d880e4fa19e439b73f903870f/users/20f9a30e0c0e422c8eedb98d46018fc9/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:44:54.013348 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57144582-c7a7-49e0-9d9d-7be2c6f64896 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:54.014035 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57144582-c7a7-49e0-9d9d-7be2c6f64896 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:54.014684 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57144582-c7a7-49e0-9d9d-7be2c6f64896 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:54.062424 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-57144582-c7a7-49e0-9d9d-7be2c6f64896 None None] MFA Rules not processed for user `20f9a30e0c0e422c8eedb98d46018fc9`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:44:54.064002 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-57144582-c7a7-49e0-9d9d-7be2c6f64896 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:44:54.093218 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5331ca-ad64-45ce-9867-1313e03b63a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:54.094676 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5331ca-ad64-45ce-9867-1313e03b63a5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:54.095047 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5331ca-ad64-45ce-9867-1313e03b63a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:54.095585 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5331ca-ad64-45ce-9867-1313e03b63a5 admin admin] PATH_INFO: `/v3/projects/885569d8959b4395b2021234c2bda1d4/users/c299a1cf60574b2c8c34e2f9f4d5e4c9/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:54.116562 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f5331ca-ad64-45ce-9867-1313e03b63a5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=885569d8959b4395b2021234c2bda1d4, user_id=c299a1cf60574b2c8c34e2f9f4d5e4c9, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:54.120335 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f5331ca-ad64-45ce-9867-1313e03b63a5 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:54.146149 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 827/1629] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:44:53 2022] PUT /identity/v3/projects/885569d8959b4395b2021234c2bda1d4/users/c299a1cf60574b2c8c34e2f9f4d5e4c9/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:44:54.147997 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5331ca-ad64-45ce-9867-1313e03b63a5 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:54.194642 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 803/1630] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:44:54 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:44:54.208410 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57144582-c7a7-49e0-9d9d-7be2c6f64896 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:54.235340 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:44:54.239022 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:44:54.266374 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5331ca-ad64-45ce-9867-1313e03b63a5 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:54.283645 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9acff9-a0b3-4d2e-bcc1-e21b6e093cf2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:54.285657 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9acff9-a0b3-4d2e-bcc1-e21b6e093cf2 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:54.285950 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9acff9-a0b3-4d2e-bcc1-e21b6e093cf2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:54.286230 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9acff9-a0b3-4d2e-bcc1-e21b6e093cf2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:54.287002 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d9acff9-a0b3-4d2e-bcc1-e21b6e093cf2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:54.288896 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d9acff9-a0b3-4d2e-bcc1-e21b6e093cf2 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:54.309282 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 828/1631] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:44:54 2022] GET /identity/v3/roles => generated 1420 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:44:54.310209 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9acff9-a0b3-4d2e-bcc1-e21b6e093cf2 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:54.325875 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9acff9-a0b3-4d2e-bcc1-e21b6e093cf2 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:54.333127 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbcbae31-f098-41d6-b71e-21a20ee46db3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:54.334129 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcbae31-f098-41d6-b71e-21a20ee46db3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:54.335295 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcbae31-f098-41d6-b71e-21a20ee46db3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:54.336292 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbcbae31-f098-41d6-b71e-21a20ee46db3 admin admin] PATH_INFO: `/v3/projects/885569d8959b4395b2021234c2bda1d4/users/c299a1cf60574b2c8c34e2f9f4d5e4c9/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:54.338862 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbcbae31-f098-41d6-b71e-21a20ee46db3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=885569d8959b4395b2021234c2bda1d4, user_id=c299a1cf60574b2c8c34e2f9f4d5e4c9, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:54.341716 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbcbae31-f098-41d6-b71e-21a20ee46db3 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:54.343096 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57144582-c7a7-49e0-9d9d-7be2c6f64896 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:54.367666 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 829/1632] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:44:54 2022] PUT /identity/v3/projects/885569d8959b4395b2021234c2bda1d4/users/c299a1cf60574b2c8c34e2f9f4d5e4c9/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:44:54.369731 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a447501-7972-45de-af6b-f3f857fc0e77 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:54.370034 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a447501-7972-45de-af6b-f3f857fc0e77 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:54.370312 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a447501-7972-45de-af6b-f3f857fc0e77 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:54.372998 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 830/1633] 10.222.0.25 () {54 vars in 912 bytes} [Fri Oct 28 09:44:54 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:44:54.374636 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7685ec-bf4f-4af0-9837-dcc2dae7f724 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:54.374969 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7685ec-bf4f-4af0-9837-dcc2dae7f724 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:54.375342 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7685ec-bf4f-4af0-9837-dcc2dae7f724 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:54.418149 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-ac7685ec-bf4f-4af0-9837-dcc2dae7f724 None None] MFA Rules not processed for user `c299a1cf60574b2c8c34e2f9f4d5e4c9`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:44:54.420414 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ac7685ec-bf4f-4af0-9837-dcc2dae7f724 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:44:54.464863 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae65374c-2065-48d4-b6bd-ec0e596b4391 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:54.465510 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae65374c-2065-48d4-b6bd-ec0e596b4391 service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:54.465799 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae65374c-2065-48d4-b6bd-ec0e596b4391 service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:54.466075 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae65374c-2065-48d4-b6bd-ec0e596b4391 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:54.546442 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 831/1634] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:44:54 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:44:54.546442 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1259723c-6fad-45da-a88c-1082b010e7c5 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:54.546442 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1259723c-6fad-45da-a88c-1082b010e7c5 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:54.547087 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1259723c-6fad-45da-a88c-1082b010e7c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:54.588352 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae65374c-2065-48d4-b6bd-ec0e596b4391 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:54.589912 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae65374c-2065-48d4-b6bd-ec0e596b4391 service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:54.593469 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-1259723c-6fad-45da-a88c-1082b010e7c5 None None] MFA Rules not processed for user `688ceb201d304c2da5373c3cf36c2d79`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:44:54.595134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-1259723c-6fad-45da-a88c-1082b010e7c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:44:54.604239 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 804/1635] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:44:54 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 396 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:44:54.605160 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae65374c-2065-48d4-b6bd-ec0e596b4391 service cinder] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:54.615883 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae65374c-2065-48d4-b6bd-ec0e596b4391 service cinder] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:54.713612 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10537758-51f4-43ae-8365-ccc6dcfba269 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:54.714333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10537758-51f4-43ae-8365-ccc6dcfba269 service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:54.714729 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10537758-51f4-43ae-8365-ccc6dcfba269 service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:54.715019 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10537758-51f4-43ae-8365-ccc6dcfba269 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:54.754849 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10537758-51f4-43ae-8365-ccc6dcfba269 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:54.757068 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10537758-51f4-43ae-8365-ccc6dcfba269 service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:54.778099 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 832/1636] 10.222.0.25 () {58 vars in 1009 bytes} [Fri Oct 28 09:44:54 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 233 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:44:54.778883 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 805/1637] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:44:54 2022] GET /identity/v3/auth/tokens => generated 3349 bytes in 174 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:44:54.822716 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1259723c-6fad-45da-a88c-1082b010e7c5 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:54.840961 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1259723c-6fad-45da-a88c-1082b010e7c5 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:54.845518 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cb187aa-ef22-4d96-8ac4-cf9903aed8b2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:54.846323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb187aa-ef22-4d96-8ac4-cf9903aed8b2 service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:54.846668 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb187aa-ef22-4d96-8ac4-cf9903aed8b2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:54.847180 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cb187aa-ef22-4d96-8ac4-cf9903aed8b2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:54.860856 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cb187aa-ef22-4d96-8ac4-cf9903aed8b2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:54.862519 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cb187aa-ef22-4d96-8ac4-cf9903aed8b2 service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:54.894845 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 833/1638] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:44:54 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 61 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:44:55.319499 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10537758-51f4-43ae-8365-ccc6dcfba269 service cinder] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:55.374994 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:44:55.506295 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10537758-51f4-43ae-8365-ccc6dcfba269 service cinder] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:55.521811 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8359a245-0d72-46ed-9036-085a62aa0c49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:55.523363 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8359a245-0d72-46ed-9036-085a62aa0c49 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:55.523896 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8359a245-0d72-46ed-9036-085a62aa0c49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:55.524280 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8359a245-0d72-46ed-9036-085a62aa0c49 admin admin] PATH_INFO: `/v3/users/92fd6faa43c44bfba111514ea3453e7d` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:55.527702 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8359a245-0d72-46ed-9036-085a62aa0c49 admin admin] RBAC: Authorizing `identity:delete_user(user_id=92fd6faa43c44bfba111514ea3453e7d)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:55.529814 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8359a245-0d72-46ed-9036-085a62aa0c49 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:55.626610 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-8359a245-0d72-46ed-9036-085a62aa0c49 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '92fd6faa43c44bfba111514ea3453e7d'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:44:55.641277 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-8359a245-0d72-46ed-9036-085a62aa0c49 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '92fd6faa43c44bfba111514ea3453e7d'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:44:55.641987 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-8359a245-0d72-46ed-9036-085a62aa0c49 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '92fd6faa43c44bfba111514ea3453e7d'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:44:55.642759 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-8359a245-0d72-46ed-9036-085a62aa0c49 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '92fd6faa43c44bfba111514ea3453e7d'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:44:55.660318 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-8359a245-0d72-46ed-9036-085a62aa0c49 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '92fd6faa43c44bfba111514ea3453e7d'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:44:55.704489 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 806/1639] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:44:55 2022] DELETE /identity/v3/users/92fd6faa43c44bfba111514ea3453e7d => generated 0 bytes in 381 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:44:55.707040 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5733dd-c57f-4984-ba41-e415cfb1ad8c None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:55.707462 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5733dd-c57f-4984-ba41-e415cfb1ad8c None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:55.707772 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5733dd-c57f-4984-ba41-e415cfb1ad8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:55.754379 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-ae5733dd-c57f-4984-ba41-e415cfb1ad8c None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:44:55.754379 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ae5733dd-c57f-4984-ba41-e415cfb1ad8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:44:55.825506 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:44:55.888478 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 834/1640] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:44:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:44:55.891760 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8359a245-0d72-46ed-9036-085a62aa0c49 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:55.894647 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-8359a245-0d72-46ed-9036-085a62aa0c49 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:44:55.923966 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5733dd-c57f-4984-ba41-e415cfb1ad8c None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:55.925573 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ae5733dd-c57f-4984-ba41-e415cfb1ad8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:44:55.926834 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8359a245-0d72-46ed-9036-085a62aa0c49 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:55.958790 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7310e346-4572-4e57-a65b-0236149bd7b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:55.958790 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7310e346-4572-4e57-a65b-0236149bd7b7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:55.958790 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7310e346-4572-4e57-a65b-0236149bd7b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:55.962004 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7310e346-4572-4e57-a65b-0236149bd7b7 admin admin] PATH_INFO: `/v3/users/d9e9bb6b29814e4d8a3e7bdeec4bec58` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:55.970707 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7310e346-4572-4e57-a65b-0236149bd7b7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d9e9bb6b29814e4d8a3e7bdeec4bec58)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:55.973128 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7310e346-4572-4e57-a65b-0236149bd7b7 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:56.054366 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5733dd-c57f-4984-ba41-e415cfb1ad8c None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:56.063054 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6793a9a-87ba-4c1d-86ef-888d4e708093 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:56.063702 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6793a9a-87ba-4c1d-86ef-888d4e708093 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:56.063989 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6793a9a-87ba-4c1d-86ef-888d4e708093 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:56.064266 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6793a9a-87ba-4c1d-86ef-888d4e708093 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:56.088076 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6793a9a-87ba-4c1d-86ef-888d4e708093 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:56.089534 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6793a9a-87ba-4c1d-86ef-888d4e708093 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:56.095611 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-7310e346-4572-4e57-a65b-0236149bd7b7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd9e9bb6b29814e4d8a3e7bdeec4bec58'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:44:56.109605 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 835/1641] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:44:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:44:56.114993 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-7310e346-4572-4e57-a65b-0236149bd7b7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd9e9bb6b29814e4d8a3e7bdeec4bec58'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:44:56.116754 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-7310e346-4572-4e57-a65b-0236149bd7b7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd9e9bb6b29814e4d8a3e7bdeec4bec58'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:44:56.119353 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-7310e346-4572-4e57-a65b-0236149bd7b7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd9e9bb6b29814e4d8a3e7bdeec4bec58'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:44:56.139453 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-7310e346-4572-4e57-a65b-0236149bd7b7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd9e9bb6b29814e4d8a3e7bdeec4bec58'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:44:56.186141 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 807/1642] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:44:55 2022] DELETE /identity/v3/users/d9e9bb6b29814e4d8a3e7bdeec4bec58 => generated 0 bytes in 294 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:44:56.189923 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd648c9-269c-4626-968a-03c9af5668ab None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:56.190220 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd648c9-269c-4626-968a-03c9af5668ab None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:56.190532 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd648c9-269c-4626-968a-03c9af5668ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:56.226759 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-ffd648c9-269c-4626-968a-03c9af5668ab None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:44:56.229256 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ffd648c9-269c-4626-968a-03c9af5668ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:44:56.365425 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 836/1643] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:44:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:44:56.384068 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec9c4f7-bb12-4db6-9396-1fad6f623cf7 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:56.384573 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec9c4f7-bb12-4db6-9396-1fad6f623cf7 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:56.384894 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec9c4f7-bb12-4db6-9396-1fad6f623cf7 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:56.389444 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 808/1644] 10.222.0.25 () {54 vars in 960 bytes} [Fri Oct 28 09:44:56 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:44:56.397996 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3febaeb7-069c-4b2a-8454-175f74e27248 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:56.398288 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3febaeb7-069c-4b2a-8454-175f74e27248 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:56.398595 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3febaeb7-069c-4b2a-8454-175f74e27248 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:56.424076 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-3febaeb7-069c-4b2a-8454-175f74e27248 None None] MFA Rules not processed for user `688ceb201d304c2da5373c3cf36c2d79`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:44:56.425493 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-3febaeb7-069c-4b2a-8454-175f74e27248 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:44:56.484181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec9c4f7-bb12-4db6-9396-1fad6f623cf7 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:56.487080 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-dec9c4f7-bb12-4db6-9396-1fad6f623cf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:44:56.516054 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:44:56.538254 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 837/1645] 10.222.0.25 () {58 vars in 1057 bytes} [Fri Oct 28 09:44:56 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:44:56.550216 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4fd9b93-66f5-4b67-9e16-e199aeea52da None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:56.551306 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4fd9b93-66f5-4b67-9e16-e199aeea52da None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:56.551665 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4fd9b93-66f5-4b67-9e16-e199aeea52da None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:56.573823 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 838/1646] 10.222.0.25 () {54 vars in 912 bytes} [Fri Oct 28 09:44:56 2022] GET /identity => generated 270 bytes in 24 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 09:44:56.583287 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bad009-8c43-4226-b9a6-48f9b95a2bb3 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:56.583760 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bad009-8c43-4226-b9a6-48f9b95a2bb3 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:56.584136 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bad009-8c43-4226-b9a6-48f9b95a2bb3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:56.620862 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-32bad009-8c43-4226-b9a6-48f9b95a2bb3 None None] MFA Rules not processed for user `688ceb201d304c2da5373c3cf36c2d79`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:44:56.623952 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-32bad009-8c43-4226-b9a6-48f9b95a2bb3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:44:56.625524 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec9c4f7-bb12-4db6-9396-1fad6f623cf7 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:56.632295 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fdd5a7b-27cf-408a-af05-b30e7e45bdec service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:56.633599 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fdd5a7b-27cf-408a-af05-b30e7e45bdec service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:56.634231 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fdd5a7b-27cf-408a-af05-b30e7e45bdec service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:56.634656 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fdd5a7b-27cf-408a-af05-b30e7e45bdec service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:56.645496 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fdd5a7b-27cf-408a-af05-b30e7e45bdec service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:56.647185 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fdd5a7b-27cf-408a-af05-b30e7e45bdec service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:56.660079 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 809/1647] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:44:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:44:56.662673 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 839/1648] 10.222.0.25 () {58 vars in 1009 bytes} [Fri Oct 28 09:44:56 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:44:56.689228 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fdd5a7b-27cf-408a-af05-b30e7e45bdec service neutron] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:56.690299 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-0fdd5a7b-27cf-408a-af05-b30e7e45bdec service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:44:56.818485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fdd5a7b-27cf-408a-af05-b30e7e45bdec service neutron] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:56.826529 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e971be48-2bf7-4dee-9163-51b7e9071043 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:56.828330 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e971be48-2bf7-4dee-9163-51b7e9071043 service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:56.828964 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e971be48-2bf7-4dee-9163-51b7e9071043 service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:56.829470 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e971be48-2bf7-4dee-9163-51b7e9071043 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:56.849061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e971be48-2bf7-4dee-9163-51b7e9071043 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:56.856148 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e971be48-2bf7-4dee-9163-51b7e9071043 service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:56.879394 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 810/1649] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:44:56 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 192 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:44:57.850144 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32bad009-8c43-4226-b9a6-48f9b95a2bb3 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:57.854343 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-32bad009-8c43-4226-b9a6-48f9b95a2bb3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:44:57.876715 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32bad009-8c43-4226-b9a6-48f9b95a2bb3 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:57.885657 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7eabc3-ce8c-4424-a659-1c3dbfa6c650 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:57.887650 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7eabc3-ce8c-4424-a659-1c3dbfa6c650 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:57.888576 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7eabc3-ce8c-4424-a659-1c3dbfa6c650 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:57.888576 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7eabc3-ce8c-4424-a659-1c3dbfa6c650 admin admin] PATH_INFO: `/v3/projects/123dc2248dae40edbaa0d87ceabfbe1e` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:57.916513 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d7eabc3-ce8c-4424-a659-1c3dbfa6c650 admin admin] RBAC: Authorizing `identity:delete_project(project_id=123dc2248dae40edbaa0d87ceabfbe1e)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:57.924034 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d7eabc3-ce8c-4424-a659-1c3dbfa6c650 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:57.929139 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:44:58.058616 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-5d7eabc3-ce8c-4424-a659-1c3dbfa6c650 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '123dc2248dae40edbaa0d87ceabfbe1e', 'request_id': 'req-5d7eabc3-ce8c-4424-a659-1c3dbfa6c650'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:44:58.070602 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-5d7eabc3-ce8c-4424-a659-1c3dbfa6c650 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '123dc2248dae40edbaa0d87ceabfbe1e', 'request_id': 'req-5d7eabc3-ce8c-4424-a659-1c3dbfa6c650'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:44:58.089722 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-5d7eabc3-ce8c-4424-a659-1c3dbfa6c650 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '123dc2248dae40edbaa0d87ceabfbe1e', 'request_id': 'req-5d7eabc3-ce8c-4424-a659-1c3dbfa6c650'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:44:58.115553 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-5d7eabc3-ce8c-4424-a659-1c3dbfa6c650 admin admin] The token cache is being invalidate because project 123dc2248dae40edbaa0d87ceabfbe1e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:44:58.116047 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-5d7eabc3-ce8c-4424-a659-1c3dbfa6c650 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:44:58.116875 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-5d7eabc3-ce8c-4424-a659-1c3dbfa6c650 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:44:58.121852 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 840/1650] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:44:57 2022] DELETE /identity/v3/projects/123dc2248dae40edbaa0d87ceabfbe1e => generated 0 bytes in 269 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:44:58.123573 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e971be48-2bf7-4dee-9163-51b7e9071043 service cinder] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:58.136895 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e971be48-2bf7-4dee-9163-51b7e9071043 service cinder] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:58.184739 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7eabc3-ce8c-4424-a659-1c3dbfa6c650 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:58.186657 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-5d7eabc3-ce8c-4424-a659-1c3dbfa6c650 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:44:58.227544 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1105bdd-a2b0-4a63-97cd-d67283675418 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:58.228986 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1105bdd-a2b0-4a63-97cd-d67283675418 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:58.229450 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1105bdd-a2b0-4a63-97cd-d67283675418 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:58.229870 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1105bdd-a2b0-4a63-97cd-d67283675418 admin admin] PATH_INFO: `/v3/users/c299a1cf60574b2c8c34e2f9f4d5e4c9` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:58.231773 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1105bdd-a2b0-4a63-97cd-d67283675418 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c299a1cf60574b2c8c34e2f9f4d5e4c9)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:58.233882 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1105bdd-a2b0-4a63-97cd-d67283675418 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:58.322026 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-b1105bdd-a2b0-4a63-97cd-d67283675418 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c299a1cf60574b2c8c34e2f9f4d5e4c9'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:44:58.331007 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7eabc3-ce8c-4424-a659-1c3dbfa6c650 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:58.338624 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-b1105bdd-a2b0-4a63-97cd-d67283675418 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c299a1cf60574b2c8c34e2f9f4d5e4c9'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:44:58.339304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-b1105bdd-a2b0-4a63-97cd-d67283675418 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c299a1cf60574b2c8c34e2f9f4d5e4c9'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:44:58.342690 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-b1105bdd-a2b0-4a63-97cd-d67283675418 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c299a1cf60574b2c8c34e2f9f4d5e4c9'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:44:58.347054 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33488714-ad9f-4867-95d6-10165d913ac6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:58.348266 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33488714-ad9f-4867-95d6-10165d913ac6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:58.350625 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:44:58.351840 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33488714-ad9f-4867-95d6-10165d913ac6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:58.352303 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33488714-ad9f-4867-95d6-10165d913ac6 admin admin] PATH_INFO: `/v3/projects/385349b18e6f4217adbcec1fb43b1e6b` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:58.354347 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33488714-ad9f-4867-95d6-10165d913ac6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=385349b18e6f4217adbcec1fb43b1e6b)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:58.357533 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33488714-ad9f-4867-95d6-10165d913ac6 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:58.359292 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-b1105bdd-a2b0-4a63-97cd-d67283675418 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c299a1cf60574b2c8c34e2f9f4d5e4c9'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:44:58.395405 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 811/1651] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:44:58 2022] DELETE /identity/v3/users/c299a1cf60574b2c8c34e2f9f4d5e4c9 => generated 0 bytes in 273 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:44:58.440137 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-33488714-ad9f-4867-95d6-10165d913ac6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '385349b18e6f4217adbcec1fb43b1e6b', 'request_id': 'req-33488714-ad9f-4867-95d6-10165d913ac6'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:44:58.452883 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-33488714-ad9f-4867-95d6-10165d913ac6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '385349b18e6f4217adbcec1fb43b1e6b', 'request_id': 'req-33488714-ad9f-4867-95d6-10165d913ac6'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:44:58.464993 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-33488714-ad9f-4867-95d6-10165d913ac6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '385349b18e6f4217adbcec1fb43b1e6b', 'request_id': 'req-33488714-ad9f-4867-95d6-10165d913ac6'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:44:58.479148 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-33488714-ad9f-4867-95d6-10165d913ac6 admin admin] The token cache is being invalidate because project 385349b18e6f4217adbcec1fb43b1e6b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:44:58.479810 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-33488714-ad9f-4867-95d6-10165d913ac6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:44:58.480659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-33488714-ad9f-4867-95d6-10165d913ac6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:44:58.482719 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 841/1652] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:44:58 2022] DELETE /identity/v3/projects/385349b18e6f4217adbcec1fb43b1e6b => generated 0 bytes in 298 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:44:58.486483 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1105bdd-a2b0-4a63-97cd-d67283675418 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:44:58.489196 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-b1105bdd-a2b0-4a63-97cd-d67283675418 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:44:58.644766 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1105bdd-a2b0-4a63-97cd-d67283675418 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:44:58.872915 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18ab2dc-e842-42a5-8139-5411f1928c04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:44:58.873570 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18ab2dc-e842-42a5-8139-5411f1928c04 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:44:58.873861 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18ab2dc-e842-42a5-8139-5411f1928c04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:44:58.874136 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18ab2dc-e842-42a5-8139-5411f1928c04 admin admin] PATH_INFO: `/v3/users/20f9a30e0c0e422c8eedb98d46018fc9` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:44:58.881979 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b18ab2dc-e842-42a5-8139-5411f1928c04 admin admin] RBAC: Authorizing `identity:delete_user(user_id=20f9a30e0c0e422c8eedb98d46018fc9)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:44:58.883020 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b18ab2dc-e842-42a5-8139-5411f1928c04 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:44:59.057439 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-b18ab2dc-e842-42a5-8139-5411f1928c04 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '20f9a30e0c0e422c8eedb98d46018fc9'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:44:59.077057 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-b18ab2dc-e842-42a5-8139-5411f1928c04 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '20f9a30e0c0e422c8eedb98d46018fc9'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:44:59.078652 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-b18ab2dc-e842-42a5-8139-5411f1928c04 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '20f9a30e0c0e422c8eedb98d46018fc9'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:44:59.080319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-b18ab2dc-e842-42a5-8139-5411f1928c04 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '20f9a30e0c0e422c8eedb98d46018fc9'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:44:59.093559 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-b18ab2dc-e842-42a5-8139-5411f1928c04 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '20f9a30e0c0e422c8eedb98d46018fc9'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:44:59.126301 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 812/1653] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:44:58 2022] DELETE /identity/v3/users/20f9a30e0c0e422c8eedb98d46018fc9 => generated 0 bytes in 640 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:45:00.676888 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33488714-ad9f-4867-95d6-10165d913ac6 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:00.683976 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-33488714-ad9f-4867-95d6-10165d913ac6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:00.737057 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33488714-ad9f-4867-95d6-10165d913ac6 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:00.827000 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:00.915134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bca21be-d704-413d-b8f3-5e586998ec0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:00.916140 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bca21be-d704-413d-b8f3-5e586998ec0c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:00.916488 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bca21be-d704-413d-b8f3-5e586998ec0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:00.916835 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bca21be-d704-413d-b8f3-5e586998ec0c admin admin] PATH_INFO: `/v3/projects/885569d8959b4395b2021234c2bda1d4` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:00.919267 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bca21be-d704-413d-b8f3-5e586998ec0c admin admin] RBAC: Authorizing `identity:delete_project(project_id=885569d8959b4395b2021234c2bda1d4)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:00.924354 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bca21be-d704-413d-b8f3-5e586998ec0c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:00.949363 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18ab2dc-e842-42a5-8139-5411f1928c04 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:00.950956 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-b18ab2dc-e842-42a5-8139-5411f1928c04 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:00.968195 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:00.980704 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18ab2dc-e842-42a5-8139-5411f1928c04 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:00.990213 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d84b58f-dd75-46b1-8103-64eff5520af6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:00.992123 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d84b58f-dd75-46b1-8103-64eff5520af6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:00.992885 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d84b58f-dd75-46b1-8103-64eff5520af6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:00.994447 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d84b58f-dd75-46b1-8103-64eff5520af6 admin admin] PATH_INFO: `/v3/projects/0f3ccd6d880e4fa19e439b73f903870f` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:01.032137 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d84b58f-dd75-46b1-8103-64eff5520af6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0f3ccd6d880e4fa19e439b73f903870f)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:01.034907 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: WARNING py.warnings [None req-9d84b58f-dd75-46b1-8103-64eff5520af6 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 28 09:45:01.034907 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: warnings.warn(msg) Oct 28 09:45:01.034907 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: Oct 28 09:45:01.043504 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d84b58f-dd75-46b1-8103-64eff5520af6 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:01.102694 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-3bca21be-d704-413d-b8f3-5e586998ec0c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '885569d8959b4395b2021234c2bda1d4', 'request_id': 'req-3bca21be-d704-413d-b8f3-5e586998ec0c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:01.122961 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-3bca21be-d704-413d-b8f3-5e586998ec0c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '885569d8959b4395b2021234c2bda1d4', 'request_id': 'req-3bca21be-d704-413d-b8f3-5e586998ec0c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:01.141877 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-3bca21be-d704-413d-b8f3-5e586998ec0c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '885569d8959b4395b2021234c2bda1d4', 'request_id': 'req-3bca21be-d704-413d-b8f3-5e586998ec0c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:01.177506 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-3bca21be-d704-413d-b8f3-5e586998ec0c admin admin] The token cache is being invalidate because project 885569d8959b4395b2021234c2bda1d4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:45:01.178051 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-3bca21be-d704-413d-b8f3-5e586998ec0c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:01.182039 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-3bca21be-d704-413d-b8f3-5e586998ec0c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:01.191554 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 842/1654] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:45:00 2022] DELETE /identity/v3/projects/885569d8959b4395b2021234c2bda1d4 => generated 0 bytes in 515 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:45:01.197734 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a101c97-146a-4664-8a99-75ac3c2156fa None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:01.197949 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a101c97-146a-4664-8a99-75ac3c2156fa None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:01.198224 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a101c97-146a-4664-8a99-75ac3c2156fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:01.284204 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-8a101c97-146a-4664-8a99-75ac3c2156fa None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:01.285481 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-8a101c97-146a-4664-8a99-75ac3c2156fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:01.296615 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-9d84b58f-dd75-46b1-8103-64eff5520af6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0f3ccd6d880e4fa19e439b73f903870f', 'request_id': 'req-9d84b58f-dd75-46b1-8103-64eff5520af6'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:01.331044 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-9d84b58f-dd75-46b1-8103-64eff5520af6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0f3ccd6d880e4fa19e439b73f903870f', 'request_id': 'req-9d84b58f-dd75-46b1-8103-64eff5520af6'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:01.353276 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-9d84b58f-dd75-46b1-8103-64eff5520af6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0f3ccd6d880e4fa19e439b73f903870f', 'request_id': 'req-9d84b58f-dd75-46b1-8103-64eff5520af6'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:01.379643 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-9d84b58f-dd75-46b1-8103-64eff5520af6 admin admin] The token cache is being invalidate because project 0f3ccd6d880e4fa19e439b73f903870f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:45:01.379935 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-9d84b58f-dd75-46b1-8103-64eff5520af6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:01.382522 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-9d84b58f-dd75-46b1-8103-64eff5520af6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:01.388585 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 813/1655] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:45:00 2022] DELETE /identity/v3/projects/0f3ccd6d880e4fa19e439b73f903870f => generated 0 bytes in 440 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:45:01.397948 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2fe30d0-b295-437b-9e06-cec0ec86eb97 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:01.398228 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2fe30d0-b295-437b-9e06-cec0ec86eb97 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:01.398576 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2fe30d0-b295-437b-9e06-cec0ec86eb97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:01.474974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-c2fe30d0-b295-437b-9e06-cec0ec86eb97 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:01.477036 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-c2fe30d0-b295-437b-9e06-cec0ec86eb97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:01.482357 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 843/1656] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 286 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:45:01.491929 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2a9dc6-d87f-40fe-b213-6e0190b88e5b None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:01.492172 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2a9dc6-d87f-40fe-b213-6e0190b88e5b None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:01.492478 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2a9dc6-d87f-40fe-b213-6e0190b88e5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:01.521641 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 814/1657] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:45:01.538617 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3edea8b-a1de-4988-910a-955b985a96ec None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:01.538904 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3edea8b-a1de-4988-910a-955b985a96ec None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:01.539176 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3edea8b-a1de-4988-910a-955b985a96ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:01.555677 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-0b2a9dc6-d87f-40fe-b213-6e0190b88e5b None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:01.559201 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-0b2a9dc6-d87f-40fe-b213-6e0190b88e5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:01.578838 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 844/1658] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:45:01.586758 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2a9dc6-d87f-40fe-b213-6e0190b88e5b None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:01.612360 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-f3edea8b-a1de-4988-910a-955b985a96ec None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:01.613612 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-f3edea8b-a1de-4988-910a-955b985a96ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:01.635875 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2a9dc6-d87f-40fe-b213-6e0190b88e5b None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:01.644841 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fba7c40-547b-4772-8d4c-9ae7bfc15c16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:01.646495 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fba7c40-547b-4772-8d4c-9ae7bfc15c16 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:01.647622 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fba7c40-547b-4772-8d4c-9ae7bfc15c16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:01.648534 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fba7c40-547b-4772-8d4c-9ae7bfc15c16 admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:01.649519 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 815/1659] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:45:01.650488 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fba7c40-547b-4772-8d4c-9ae7bfc15c16 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:45:01.650998 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fba7c40-547b-4772-8d4c-9ae7bfc15c16 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:01.653286 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fba7c40-547b-4772-8d4c-9ae7bfc15c16 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:01.654962 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3edea8b-a1de-4988-910a-955b985a96ec None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:01.682078 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 845/1660] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:45:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:45:01.685464 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3edea8b-a1de-4988-910a-955b985a96ec None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:01.687807 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fba7c40-547b-4772-8d4c-9ae7bfc15c16 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:01.703189 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fba7c40-547b-4772-8d4c-9ae7bfc15c16 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:01.711197 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff66622b-ecde-441b-b9a1-360306ce5cc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:01.711351 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8c0137-0269-4368-a54d-fba33faade09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:01.712203 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8c0137-0269-4368-a54d-fba33faade09 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:01.712484 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8c0137-0269-4368-a54d-fba33faade09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:01.712752 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8c0137-0269-4368-a54d-fba33faade09 admin admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:01.713425 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e8c0137-0269-4368-a54d-fba33faade09 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:45:01.713731 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e8c0137-0269-4368-a54d-fba33faade09 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:01.713900 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff66622b-ecde-441b-b9a1-360306ce5cc4 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:01.714195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff66622b-ecde-441b-b9a1-360306ce5cc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:01.714576 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff66622b-ecde-441b-b9a1-360306ce5cc4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:01.715228 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff66622b-ecde-441b-b9a1-360306ce5cc4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:01.716172 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e8c0137-0269-4368-a54d-fba33faade09 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:01.718201 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff66622b-ecde-441b-b9a1-360306ce5cc4 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:01.770447 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 816/1661] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:45:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 117 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:45:01.775067 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 846/1662] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:45:01 2022] POST /identity/v3/projects => generated 375 bytes in 88 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:45:01.775219 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8c0137-0269-4368-a54d-fba33faade09 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:01.781290 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff66622b-ecde-441b-b9a1-360306ce5cc4 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:01.799793 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8c0137-0269-4368-a54d-fba33faade09 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:01.828971 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:01.961869 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff66622b-ecde-441b-b9a1-360306ce5cc4 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:01.967928 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe782cc-039f-48ed-9cca-443f8daf12ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:01.969890 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe782cc-039f-48ed-9cca-443f8daf12ef admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:01.970361 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe782cc-039f-48ed-9cca-443f8daf12ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:01.970893 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe782cc-039f-48ed-9cca-443f8daf12ef admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:01.971708 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe782cc-039f-48ed-9cca-443f8daf12ef admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:01.974014 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe782cc-039f-48ed-9cca-443f8daf12ef admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:02.006842 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d264b2-a10c-41b6-9c5f-f901101094d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:02.007654 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d264b2-a10c-41b6-9c5f-f901101094d0 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:02.007853 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d264b2-a10c-41b6-9c5f-f901101094d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:02.008004 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d264b2-a10c-41b6-9c5f-f901101094d0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:02.008655 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d264b2-a10c-41b6-9c5f-f901101094d0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:02.010624 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d264b2-a10c-41b6-9c5f-f901101094d0 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:02.070973 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 847/1663] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:45:01 2022] POST /identity/v3/users => generated 288 bytes in 292 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:45:02.072370 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe782cc-039f-48ed-9cca-443f8daf12ef admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:02.088673 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe782cc-039f-48ed-9cca-443f8daf12ef admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:02.097461 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0928684c-3ba1-4638-adbc-d3e3ca62062c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:02.097461 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0928684c-3ba1-4638-adbc-d3e3ca62062c admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:02.097461 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0928684c-3ba1-4638-adbc-d3e3ca62062c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:02.097461 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0928684c-3ba1-4638-adbc-d3e3ca62062c admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:02.097901 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0928684c-3ba1-4638-adbc-d3e3ca62062c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:02.098831 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0928684c-3ba1-4638-adbc-d3e3ca62062c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:02.105081 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 817/1664] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:45:01 2022] POST /identity/v3/projects => generated 357 bytes in 330 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:45:02.110456 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d264b2-a10c-41b6-9c5f-f901101094d0 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:02.123563 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 848/1665] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:45:02 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:45:02.124781 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0928684c-3ba1-4638-adbc-d3e3ca62062c admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:02.148408 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0928684c-3ba1-4638-adbc-d3e3ca62062c admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:02.267546 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:02.302547 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ebb2354-12a7-4fd5-98fa-c4b40f63115b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:02.303892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebb2354-12a7-4fd5-98fa-c4b40f63115b admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:02.307105 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebb2354-12a7-4fd5-98fa-c4b40f63115b admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:02.307559 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ebb2354-12a7-4fd5-98fa-c4b40f63115b admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:02.309985 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ebb2354-12a7-4fd5-98fa-c4b40f63115b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:02.314192 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ebb2354-12a7-4fd5-98fa-c4b40f63115b admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:02.317140 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d264b2-a10c-41b6-9c5f-f901101094d0 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:02.331260 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa95bba7-5a8f-44c8-adfc-f2e297f56124 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:02.331927 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa95bba7-5a8f-44c8-adfc-f2e297f56124 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:02.332212 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa95bba7-5a8f-44c8-adfc-f2e297f56124 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:02.332489 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa95bba7-5a8f-44c8-adfc-f2e297f56124 admin admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:02.333147 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa95bba7-5a8f-44c8-adfc-f2e297f56124 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:02.335691 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa95bba7-5a8f-44c8-adfc-f2e297f56124 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:02.344262 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 849/1666] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:45:02 2022] GET /identity/v3/roles => generated 1420 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:45:02.358568 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ebb2354-12a7-4fd5-98fa-c4b40f63115b admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:02.380633 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 818/1667] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:45:02 2022] POST /identity/v3/users => generated 279 bytes in 271 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:45:02.387533 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa95bba7-5a8f-44c8-adfc-f2e297f56124 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:02.405445 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ebb2354-12a7-4fd5-98fa-c4b40f63115b admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:02.418373 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa95bba7-5a8f-44c8-adfc-f2e297f56124 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:02.424009 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8e282b6-78e0-4f3b-b0cb-aa1755cce35e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:02.429318 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e282b6-78e0-4f3b-b0cb-aa1755cce35e admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:02.429877 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e282b6-78e0-4f3b-b0cb-aa1755cce35e admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:02.431493 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e282b6-78e0-4f3b-b0cb-aa1755cce35e admin admin] PATH_INFO: `/v3/projects/aa92d710b3a84607a6d958773640b398/users/39e891d5d8a6431fb6f5343c543214c5/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:02.442462 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691e7c2d-c05a-4d9a-8444-739b5ee57f2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:02.443316 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691e7c2d-c05a-4d9a-8444-739b5ee57f2e admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:02.443803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691e7c2d-c05a-4d9a-8444-739b5ee57f2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:02.444222 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691e7c2d-c05a-4d9a-8444-739b5ee57f2e admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:02.445392 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-691e7c2d-c05a-4d9a-8444-739b5ee57f2e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:02.447412 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-691e7c2d-c05a-4d9a-8444-739b5ee57f2e admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:02.470567 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 819/1668] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:45:02 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:45:02.474812 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691e7c2d-c05a-4d9a-8444-739b5ee57f2e admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:02.495774 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691e7c2d-c05a-4d9a-8444-739b5ee57f2e admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:02.496299 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8e282b6-78e0-4f3b-b0cb-aa1755cce35e admin admin] RBAC: Authorizing `identity:create_grant(project_id=aa92d710b3a84607a6d958773640b398, user_id=39e891d5d8a6431fb6f5343c543214c5, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:02.512954 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8e282b6-78e0-4f3b-b0cb-aa1755cce35e admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:02.513876 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d9d4baf-4cfd-4392-93b2-b767bb45332b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:02.514728 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9d4baf-4cfd-4392-93b2-b767bb45332b admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:02.515101 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9d4baf-4cfd-4392-93b2-b767bb45332b admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:02.515426 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9d4baf-4cfd-4392-93b2-b767bb45332b admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:02.516146 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d9d4baf-4cfd-4392-93b2-b767bb45332b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:02.525280 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d9d4baf-4cfd-4392-93b2-b767bb45332b admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:02.549858 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 820/1669] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:45:02 2022] GET /identity/v3/roles => generated 1420 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:45:02.554451 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d9d4baf-4cfd-4392-93b2-b767bb45332b admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:02.578044 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 850/1670] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:45:02 2022] PUT /identity/v3/projects/aa92d710b3a84607a6d958773640b398/users/39e891d5d8a6431fb6f5343c543214c5/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:45:02.578551 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d9d4baf-4cfd-4392-93b2-b767bb45332b admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:02.586869 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f01ad1b-109e-4171-a19f-e6e030cdbbea None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:02.587092 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f01ad1b-109e-4171-a19f-e6e030cdbbea None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:02.587328 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f01ad1b-109e-4171-a19f-e6e030cdbbea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:02.699866 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8849cd9-9991-480d-b775-40d9e17f06b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:02.700818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8849cd9-9991-480d-b775-40d9e17f06b8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:02.701298 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8849cd9-9991-480d-b775-40d9e17f06b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:02.701684 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8849cd9-9991-480d-b775-40d9e17f06b8 admin admin] PATH_INFO: `/v3/projects/d8700031698d4e7eb69fb34d2d1c73e8/users/5a4b6abb24e4425f9427d88487ad2062/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:02.744392 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-7f01ad1b-109e-4171-a19f-e6e030cdbbea None None] MFA Rules not processed for user `39e891d5d8a6431fb6f5343c543214c5`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:02.752313 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7f01ad1b-109e-4171-a19f-e6e030cdbbea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:02.753561 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8849cd9-9991-480d-b775-40d9e17f06b8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d8700031698d4e7eb69fb34d2d1c73e8, user_id=5a4b6abb24e4425f9427d88487ad2062, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:02.758025 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8849cd9-9991-480d-b775-40d9e17f06b8 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:02.783670 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 821/1671] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:45:02 2022] PUT /identity/v3/projects/d8700031698d4e7eb69fb34d2d1c73e8/users/5a4b6abb24e4425f9427d88487ad2062/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:45:02.796494 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6959e53-4c71-4091-a5dd-9ad3d3a61144 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:02.796860 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6959e53-4c71-4091-a5dd-9ad3d3a61144 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:02.797235 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6959e53-4c71-4091-a5dd-9ad3d3a61144 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:02.863518 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-c6959e53-4c71-4091-a5dd-9ad3d3a61144 None None] MFA Rules not processed for user `5a4b6abb24e4425f9427d88487ad2062`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:02.864995 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-c6959e53-4c71-4091-a5dd-9ad3d3a61144 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:02.990957 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:03.051525 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 851/1672] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:02 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 466 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:45:03.056763 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f01ad1b-109e-4171-a19f-e6e030cdbbea None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:03.113859 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f01ad1b-109e-4171-a19f-e6e030cdbbea None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:03.121677 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 822/1673] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:02 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 326 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:45:03.128788 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6959e53-4c71-4091-a5dd-9ad3d3a61144 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:03.230898 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:03.347351 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6453a504-799a-45c4-a8c2-b28f752543cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:03.348167 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6453a504-799a-45c4-a8c2-b28f752543cc admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:03.348558 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6453a504-799a-45c4-a8c2-b28f752543cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:03.349752 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6453a504-799a-45c4-a8c2-b28f752543cc admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:03.349752 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6453a504-799a-45c4-a8c2-b28f752543cc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:03.353634 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6453a504-799a-45c4-a8c2-b28f752543cc admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:03.359336 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6959e53-4c71-4091-a5dd-9ad3d3a61144 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:03.379535 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bac5f2b-7262-46f8-8881-bc67acbd0c72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:03.380304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bac5f2b-7262-46f8-8881-bc67acbd0c72 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:03.380760 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bac5f2b-7262-46f8-8881-bc67acbd0c72 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:03.382257 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bac5f2b-7262-46f8-8881-bc67acbd0c72 admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:03.383098 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bac5f2b-7262-46f8-8881-bc67acbd0c72 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:03.387912 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bac5f2b-7262-46f8-8881-bc67acbd0c72 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:03.450820 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 852/1674] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:45:03 2022] POST /identity/v3/projects => generated 377 bytes in 394 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:45:03.453257 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6453a504-799a-45c4-a8c2-b28f752543cc admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:03.488256 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 823/1675] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:45:03 2022] POST /identity/v3/projects => generated 357 bytes in 362 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:45:03.492171 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bac5f2b-7262-46f8-8881-bc67acbd0c72 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:03.664762 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6453a504-799a-45c4-a8c2-b28f752543cc admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:03.733477 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bac5f2b-7262-46f8-8881-bc67acbd0c72 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:03.746699 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0812431-1932-43e4-b0d7-93f8ddb38c13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:03.747348 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0812431-1932-43e4-b0d7-93f8ddb38c13 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:03.748080 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0812431-1932-43e4-b0d7-93f8ddb38c13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:03.748080 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0812431-1932-43e4-b0d7-93f8ddb38c13 admin admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:03.748553 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0812431-1932-43e4-b0d7-93f8ddb38c13 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:03.750638 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0812431-1932-43e4-b0d7-93f8ddb38c13 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:03.806143 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 824/1676] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:45:03 2022] POST /identity/v3/users => generated 285 bytes in 315 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:45:03.807112 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0812431-1932-43e4-b0d7-93f8ddb38c13 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:03.814090 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-c0812431-1932-43e4-b0d7-93f8ddb38c13 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:03.822910 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc466a97-c042-4e3b-a9ee-3a6db54870e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:03.824897 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc466a97-c042-4e3b-a9ee-3a6db54870e9 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:03.824897 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc466a97-c042-4e3b-a9ee-3a6db54870e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:03.824897 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc466a97-c042-4e3b-a9ee-3a6db54870e9 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:03.824897 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc466a97-c042-4e3b-a9ee-3a6db54870e9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:03.826719 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc466a97-c042-4e3b-a9ee-3a6db54870e9 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:03.841819 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0812431-1932-43e4-b0d7-93f8ddb38c13 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:03.851679 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1641eb0f-dca5-43fd-96ed-1de08023c451 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:03.852369 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1641eb0f-dca5-43fd-96ed-1de08023c451 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:03.852778 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1641eb0f-dca5-43fd-96ed-1de08023c451 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:03.852976 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1641eb0f-dca5-43fd-96ed-1de08023c451 admin admin] PATH_INFO: `/v3/users/f63cfb4896ff45848bbd79a65ef0cdf1` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:03.859160 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1641eb0f-dca5-43fd-96ed-1de08023c451 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f63cfb4896ff45848bbd79a65ef0cdf1)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:03.861443 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1641eb0f-dca5-43fd-96ed-1de08023c451 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:03.867546 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 853/1677] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:45:03 2022] POST /identity/v3/users => generated 295 bytes in 411 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:45:03.867546 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc466a97-c042-4e3b-a9ee-3a6db54870e9 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:03.889195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc466a97-c042-4e3b-a9ee-3a6db54870e9 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:03.904676 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbcf5982-bef0-42cb-b8f0-9f4f71524092 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:03.905519 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbcf5982-bef0-42cb-b8f0-9f4f71524092 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:03.905807 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbcf5982-bef0-42cb-b8f0-9f4f71524092 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:03.906084 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbcf5982-bef0-42cb-b8f0-9f4f71524092 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:03.906888 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbcf5982-bef0-42cb-b8f0-9f4f71524092 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:03.908523 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbcf5982-bef0-42cb-b8f0-9f4f71524092 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:03.932114 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 854/1678] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:45:03 2022] GET /identity/v3/roles => generated 1420 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:45:03.932114 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbcf5982-bef0-42cb-b8f0-9f4f71524092 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:03.945539 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:03.953080 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbcf5982-bef0-42cb-b8f0-9f4f71524092 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:03.973607 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f179ff5-eee0-40b4-a2c6-1e799a065caa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:03.974698 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f179ff5-eee0-40b4-a2c6-1e799a065caa admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:03.975049 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f179ff5-eee0-40b4-a2c6-1e799a065caa admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:03.976325 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f179ff5-eee0-40b4-a2c6-1e799a065caa admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:03.976325 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f179ff5-eee0-40b4-a2c6-1e799a065caa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:03.977782 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f179ff5-eee0-40b4-a2c6-1e799a065caa admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:03.983242 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-1641eb0f-dca5-43fd-96ed-1de08023c451 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f63cfb4896ff45848bbd79a65ef0cdf1'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:04.005762 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 855/1679] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:45:03 2022] GET /identity/v3/roles => generated 1420 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:45:04.006684 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f179ff5-eee0-40b4-a2c6-1e799a065caa admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:04.032330 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f179ff5-eee0-40b4-a2c6-1e799a065caa admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:04.038974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-1641eb0f-dca5-43fd-96ed-1de08023c451 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f63cfb4896ff45848bbd79a65ef0cdf1'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:04.039755 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-1641eb0f-dca5-43fd-96ed-1de08023c451 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f63cfb4896ff45848bbd79a65ef0cdf1'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:04.040470 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-1641eb0f-dca5-43fd-96ed-1de08023c451 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f63cfb4896ff45848bbd79a65ef0cdf1'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:04.052808 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d991c174-3eee-4571-815c-344054902404 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:04.054858 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d991c174-3eee-4571-815c-344054902404 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:04.055303 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d991c174-3eee-4571-815c-344054902404 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:04.056332 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d991c174-3eee-4571-815c-344054902404 admin admin] PATH_INFO: `/v3/projects/ba157963b89d4f0c928fa9bc50792d1d/users/b3838c4e1dfe40cdbcdc28964810eb4b/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:04.058121 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-1641eb0f-dca5-43fd-96ed-1de08023c451 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f63cfb4896ff45848bbd79a65ef0cdf1'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:04.124012 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 825/1680] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:45:03 2022] DELETE /identity/v3/users/f63cfb4896ff45848bbd79a65ef0cdf1 => generated 0 bytes in 317 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:45:04.124994 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1641eb0f-dca5-43fd-96ed-1de08023c451 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:04.127867 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-1641eb0f-dca5-43fd-96ed-1de08023c451 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:04.150442 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d991c174-3eee-4571-815c-344054902404 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ba157963b89d4f0c928fa9bc50792d1d, user_id=b3838c4e1dfe40cdbcdc28964810eb4b, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:04.156103 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d991c174-3eee-4571-815c-344054902404 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:04.159863 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=85,keystone:INSERT=5,keystone:DELETE=10 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:04.190640 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 856/1681] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:45:04 2022] PUT /identity/v3/projects/ba157963b89d4f0c928fa9bc50792d1d/users/b3838c4e1dfe40cdbcdc28964810eb4b/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:45:04.197983 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00546939-ab43-4772-a43e-8ffb55cc6cf5 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:04.197983 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00546939-ab43-4772-a43e-8ffb55cc6cf5 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:04.198284 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00546939-ab43-4772-a43e-8ffb55cc6cf5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:04.237153 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-00546939-ab43-4772-a43e-8ffb55cc6cf5 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:04.239263 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-00546939-ab43-4772-a43e-8ffb55cc6cf5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:04.266733 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1641eb0f-dca5-43fd-96ed-1de08023c451 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:04.419144 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba213ea-c0a6-4a20-b707-9c00344848f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:04.419361 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 857/1682] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:45:04.420185 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba213ea-c0a6-4a20-b707-9c00344848f1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:04.420630 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba213ea-c0a6-4a20-b707-9c00344848f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:04.421052 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba213ea-c0a6-4a20-b707-9c00344848f1 admin admin] PATH_INFO: `/v3/projects/7f025a2677be439197ad88613957a816/users/fbb4fbdcd1994a98b9a4966fb6adac75/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:04.423511 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00546939-ab43-4772-a43e-8ffb55cc6cf5 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:04.427494 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-00546939-ab43-4772-a43e-8ffb55cc6cf5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:04.462537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00546939-ab43-4772-a43e-8ffb55cc6cf5 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:04.467900 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba213ea-c0a6-4a20-b707-9c00344848f1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7f025a2677be439197ad88613957a816, user_id=fbb4fbdcd1994a98b9a4966fb6adac75, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:04.470939 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba213ea-c0a6-4a20-b707-9c00344848f1 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:04.483559 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc78a68b-9b5c-47f2-b0f2-e499ca3f3df7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:04.484534 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc78a68b-9b5c-47f2-b0f2-e499ca3f3df7 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:04.484833 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc78a68b-9b5c-47f2-b0f2-e499ca3f3df7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:04.485654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc78a68b-9b5c-47f2-b0f2-e499ca3f3df7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:04.486831 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc78a68b-9b5c-47f2-b0f2-e499ca3f3df7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:04.489334 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc78a68b-9b5c-47f2-b0f2-e499ca3f3df7 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:04.498835 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 826/1683] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:45:04 2022] PUT /identity/v3/projects/7f025a2677be439197ad88613957a816/users/fbb4fbdcd1994a98b9a4966fb6adac75/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 374 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:45:04.499920 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba213ea-c0a6-4a20-b707-9c00344848f1 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:04.503881 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-dba213ea-c0a6-4a20-b707-9c00344848f1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:04.518363 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 858/1684] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:45:04 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:45:04.518671 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc78a68b-9b5c-47f2-b0f2-e499ca3f3df7 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:04.558334 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc78a68b-9b5c-47f2-b0f2-e499ca3f3df7 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:04.671957 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba213ea-c0a6-4a20-b707-9c00344848f1 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:04.683257 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6efb513d-6bc5-45f9-adf6-ca694bfd5b81 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:04.684011 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6efb513d-6bc5-45f9-adf6-ca694bfd5b81 service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:04.684183 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6efb513d-6bc5-45f9-adf6-ca694bfd5b81 service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:04.684468 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6efb513d-6bc5-45f9-adf6-ca694bfd5b81 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:04.722003 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ec441b-dbfa-4f19-8937-fa148a012824 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:04.725177 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ec441b-dbfa-4f19-8937-fa148a012824 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:04.725339 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ec441b-dbfa-4f19-8937-fa148a012824 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:04.725519 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ec441b-dbfa-4f19-8937-fa148a012824 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:04.729822 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52ec441b-dbfa-4f19-8937-fa148a012824 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:04.735815 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52ec441b-dbfa-4f19-8937-fa148a012824 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:04.775110 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 859/1685] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:45:04 2022] GET /identity/v3/roles => generated 1420 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:45:04.775110 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ec441b-dbfa-4f19-8937-fa148a012824 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:04.797052 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ec441b-dbfa-4f19-8937-fa148a012824 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:04.807743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c765b4b8-c1a9-48be-8a4f-9192cee009e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:04.808870 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c765b4b8-c1a9-48be-8a4f-9192cee009e3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:04.810199 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c765b4b8-c1a9-48be-8a4f-9192cee009e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:04.810709 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c765b4b8-c1a9-48be-8a4f-9192cee009e3 admin admin] PATH_INFO: `/v3/projects/ba157963b89d4f0c928fa9bc50792d1d/users/b3838c4e1dfe40cdbcdc28964810eb4b/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:04.813946 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c765b4b8-c1a9-48be-8a4f-9192cee009e3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ba157963b89d4f0c928fa9bc50792d1d, user_id=b3838c4e1dfe40cdbcdc28964810eb4b, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:04.817189 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c765b4b8-c1a9-48be-8a4f-9192cee009e3 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:04.839275 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 860/1686] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:45:04 2022] PUT /identity/v3/projects/ba157963b89d4f0c928fa9bc50792d1d/users/b3838c4e1dfe40cdbcdc28964810eb4b/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:45:04.853730 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c765b4b8-c1a9-48be-8a4f-9192cee009e3 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:04.857000 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:04.891617 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6efb513d-6bc5-45f9-adf6-ca694bfd5b81 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:04.893119 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6efb513d-6bc5-45f9-adf6-ca694bfd5b81 service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:04.917350 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:04.920105 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 827/1687] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:45:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 420 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:45:04.921442 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e195398-f660-485b-a96f-365765c0d6e9 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:04.921738 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e195398-f660-485b-a96f-365765c0d6e9 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:04.922019 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e195398-f660-485b-a96f-365765c0d6e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:04.945622 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c765b4b8-c1a9-48be-8a4f-9192cee009e3 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:04.952823 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb78349e-6b03-4135-ade5-d79f9a095369 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:04.953616 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb78349e-6b03-4135-ade5-d79f9a095369 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:04.954004 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb78349e-6b03-4135-ade5-d79f9a095369 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:04.955513 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb78349e-6b03-4135-ade5-d79f9a095369 admin admin] PATH_INFO: `/v3/projects/7f025a2677be439197ad88613957a816/users/fbb4fbdcd1994a98b9a4966fb6adac75/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:04.957660 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb78349e-6b03-4135-ade5-d79f9a095369 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7f025a2677be439197ad88613957a816, user_id=fbb4fbdcd1994a98b9a4966fb6adac75, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:04.960647 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb78349e-6b03-4135-ade5-d79f9a095369 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:04.989223 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 861/1688] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:45:04 2022] PUT /identity/v3/projects/7f025a2677be439197ad88613957a816/users/fbb4fbdcd1994a98b9a4966fb6adac75/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:45:04.997423 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113be21f-7e0e-41ff-8027-b21867ead94b None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:04.997779 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113be21f-7e0e-41ff-8027-b21867ead94b None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:04.998068 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113be21f-7e0e-41ff-8027-b21867ead94b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:05.012301 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-4e195398-f660-485b-a96f-365765c0d6e9 None None] MFA Rules not processed for user `b3838c4e1dfe40cdbcdc28964810eb4b`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:05.014328 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-4e195398-f660-485b-a96f-365765c0d6e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:05.039454 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-113be21f-7e0e-41ff-8027-b21867ead94b None None] MFA Rules not processed for user `fbb4fbdcd1994a98b9a4966fb6adac75`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:05.041656 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-113be21f-7e0e-41ff-8027-b21867ead94b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:05.145017 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 862/1689] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:04 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:45:05.159883 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-113be21f-7e0e-41ff-8027-b21867ead94b None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:05.161296 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-113be21f-7e0e-41ff-8027-b21867ead94b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:05.219178 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 828/1690] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:04 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 298 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:45:05.232013 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e195398-f660-485b-a96f-365765c0d6e9 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:05.233629 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-4e195398-f660-485b-a96f-365765c0d6e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:05.324400 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-113be21f-7e0e-41ff-8027-b21867ead94b None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:05.340974 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba2f46f-eddc-4b83-996f-4c1864274253 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:05.342233 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba2f46f-eddc-4b83-996f-4c1864274253 service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:05.342544 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba2f46f-eddc-4b83-996f-4c1864274253 service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:05.342825 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba2f46f-eddc-4b83-996f-4c1864274253 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:05.345017 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-cba2f46f-eddc-4b83-996f-4c1864274253 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:05.346073 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e195398-f660-485b-a96f-365765c0d6e9 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:05.352120 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de48fb4d-7765-43ed-936c-300c119596ff service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:05.353100 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de48fb4d-7765-43ed-936c-300c119596ff service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:05.353524 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de48fb4d-7765-43ed-936c-300c119596ff service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:05.353935 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de48fb4d-7765-43ed-936c-300c119596ff service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:05.368746 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de48fb4d-7765-43ed-936c-300c119596ff service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:05.370823 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de48fb4d-7765-43ed-936c-300c119596ff service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:05.403820 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 829/1691] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:45:05 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 173 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:45:05.486912 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cba2f46f-eddc-4b83-996f-4c1864274253 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:05.488359 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cba2f46f-eddc-4b83-996f-4c1864274253 service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:05.522620 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 863/1692] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:45:05 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 363 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:45:05.533453 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de48fb4d-7765-43ed-936c-300c119596ff service cinder] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:05.548689 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de48fb4d-7765-43ed-936c-300c119596ff service cinder] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:05.560739 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-620251bf-f4e7-4ec0-9d40-cfc846059412 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:05.562425 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620251bf-f4e7-4ec0-9d40-cfc846059412 service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:05.562425 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620251bf-f4e7-4ec0-9d40-cfc846059412 service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:05.562425 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620251bf-f4e7-4ec0-9d40-cfc846059412 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:05.566733 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-620251bf-f4e7-4ec0-9d40-cfc846059412 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:05.702252 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-620251bf-f4e7-4ec0-9d40-cfc846059412 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:05.703746 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-620251bf-f4e7-4ec0-9d40-cfc846059412 service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:05.731525 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 830/1693] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:45:05 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 199 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:45:07.385207 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621b15f7-e52a-44ad-8996-227991d92549 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:07.387043 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621b15f7-e52a-44ad-8996-227991d92549 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:07.387487 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621b15f7-e52a-44ad-8996-227991d92549 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:07.492134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:07.506334 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-621b15f7-e52a-44ad-8996-227991d92549 None None] MFA Rules not processed for user `ed9a4c24cbfb4689b7f984219c7503a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:07.510502 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-621b15f7-e52a-44ad-8996-227991d92549 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:07.723390 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 864/1694] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:07 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 340 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:45:07.735346 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c803bc1a-d1d8-4846-8038-bb600ed1b9a6 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:07.736214 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c803bc1a-d1d8-4846-8038-bb600ed1b9a6 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:07.736793 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c803bc1a-d1d8-4846-8038-bb600ed1b9a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:07.806381 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-c803bc1a-d1d8-4846-8038-bb600ed1b9a6 None None] MFA Rules not processed for user `ed9a4c24cbfb4689b7f984219c7503a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:07.813328 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621b15f7-e52a-44ad-8996-227991d92549 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:07.815309 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-621b15f7-e52a-44ad-8996-227991d92549 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:07.816616 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-c803bc1a-d1d8-4846-8038-bb600ed1b9a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:07.847679 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 831/1695] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:07 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:45:07.858174 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c803bc1a-d1d8-4846-8038-bb600ed1b9a6 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:07.859917 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-c803bc1a-d1d8-4846-8038-bb600ed1b9a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:07.919689 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:08.013099 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621b15f7-e52a-44ad-8996-227991d92549 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:08.027621 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f936a8c1-260d-4392-a868-ffe6ab72758c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:08.033102 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f936a8c1-260d-4392-a868-ffe6ab72758c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:08.033574 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f936a8c1-260d-4392-a868-ffe6ab72758c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:08.033857 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f936a8c1-260d-4392-a868-ffe6ab72758c admin admin] PATH_INFO: `/v3/projects/1d65871da7a445fda2f75ea82af5f9b9` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:08.036603 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f936a8c1-260d-4392-a868-ffe6ab72758c admin admin] RBAC: Authorizing `identity:delete_project(project_id=1d65871da7a445fda2f75ea82af5f9b9)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:08.043908 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f936a8c1-260d-4392-a868-ffe6ab72758c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:08.065772 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c803bc1a-d1d8-4846-8038-bb600ed1b9a6 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:08.080282 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519e7a55-bf99-455f-837d-d383b94c1bb3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:08.081641 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519e7a55-bf99-455f-837d-d383b94c1bb3 service nova] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:08.082064 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519e7a55-bf99-455f-837d-d383b94c1bb3 service nova] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:08.082338 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519e7a55-bf99-455f-837d-d383b94c1bb3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:08.103197 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519e7a55-bf99-455f-837d-d383b94c1bb3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:08.105915 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519e7a55-bf99-455f-837d-d383b94c1bb3 service nova] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:08.133419 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 832/1696] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:45:07 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 276 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:45:08.135668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c80d64c-5515-4fb2-9bfc-48e61bf74b44 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:08.136223 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c80d64c-5515-4fb2-9bfc-48e61bf74b44 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:08.136449 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c80d64c-5515-4fb2-9bfc-48e61bf74b44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:08.148492 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-f936a8c1-260d-4392-a868-ffe6ab72758c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1d65871da7a445fda2f75ea82af5f9b9', 'request_id': 'req-f936a8c1-260d-4392-a868-ffe6ab72758c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:08.166580 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-f936a8c1-260d-4392-a868-ffe6ab72758c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1d65871da7a445fda2f75ea82af5f9b9', 'request_id': 'req-f936a8c1-260d-4392-a868-ffe6ab72758c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:08.183141 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-f936a8c1-260d-4392-a868-ffe6ab72758c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1d65871da7a445fda2f75ea82af5f9b9', 'request_id': 'req-f936a8c1-260d-4392-a868-ffe6ab72758c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:08.225999 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-f936a8c1-260d-4392-a868-ffe6ab72758c admin admin] The token cache is being invalidate because project 1d65871da7a445fda2f75ea82af5f9b9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:45:08.226744 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-f936a8c1-260d-4392-a868-ffe6ab72758c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:08.227186 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-2c80d64c-5515-4fb2-9bfc-48e61bf74b44 None None] MFA Rules not processed for user `715a782e5a56431e88e4fe5a3b0a62aa`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:08.228446 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-f936a8c1-260d-4392-a868-ffe6ab72758c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:08.230669 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-2c80d64c-5515-4fb2-9bfc-48e61bf74b44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:08.231911 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 865/1697] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:45:07 2022] DELETE /identity/v3/projects/1d65871da7a445fda2f75ea82af5f9b9 => generated 0 bytes in 419 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:45:08.237107 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62439ab-3dab-41d6-9eb0-3c787ad68045 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:08.237388 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62439ab-3dab-41d6-9eb0-3c787ad68045 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:08.237664 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62439ab-3dab-41d6-9eb0-3c787ad68045 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:08.302143 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-d62439ab-3dab-41d6-9eb0-3c787ad68045 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:08.304094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d62439ab-3dab-41d6-9eb0-3c787ad68045 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:08.352348 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:08.419240 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 866/1698] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:45:08.420996 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb60eba-389d-401d-beb1-88dbc52b427e None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:08.421281 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb60eba-389d-401d-beb1-88dbc52b427e None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:08.421560 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb60eba-389d-401d-beb1-88dbc52b427e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:08.436424 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 833/1699] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:08 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 302 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:45:08.445344 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c80d64c-5515-4fb2-9bfc-48e61bf74b44 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:08.447807 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-2c80d64c-5515-4fb2-9bfc-48e61bf74b44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:08.460999 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-2bb60eba-389d-401d-beb1-88dbc52b427e None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:08.462963 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-2bb60eba-389d-401d-beb1-88dbc52b427e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:08.490939 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 867/1700] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 67 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:45:08.492014 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb60eba-389d-401d-beb1-88dbc52b427e None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:08.509681 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb60eba-389d-401d-beb1-88dbc52b427e None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:08.522341 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b1962a-2f21-45a0-93aa-9bca9747fe0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:08.523005 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b1962a-2f21-45a0-93aa-9bca9747fe0b admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:08.523288 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b1962a-2f21-45a0-93aa-9bca9747fe0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:08.523565 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b1962a-2f21-45a0-93aa-9bca9747fe0b admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:08.524230 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2b1962a-2f21-45a0-93aa-9bca9747fe0b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:45:08.524536 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2b1962a-2f21-45a0-93aa-9bca9747fe0b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:08.535064 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2b1962a-2f21-45a0-93aa-9bca9747fe0b admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:08.594753 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 868/1701] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:45:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 103 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:45:08.595730 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b1962a-2f21-45a0-93aa-9bca9747fe0b admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:08.628183 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b1962a-2f21-45a0-93aa-9bca9747fe0b admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:08.652899 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a17de8ce-2700-4cf7-89bf-83b9897f2a46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:08.658772 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a17de8ce-2700-4cf7-89bf-83b9897f2a46 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:08.659142 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a17de8ce-2700-4cf7-89bf-83b9897f2a46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:08.659582 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a17de8ce-2700-4cf7-89bf-83b9897f2a46 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:08.660268 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a17de8ce-2700-4cf7-89bf-83b9897f2a46 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:08.662184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a17de8ce-2700-4cf7-89bf-83b9897f2a46 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:08.713861 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c80d64c-5515-4fb2-9bfc-48e61bf74b44 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:08.747671 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b6511a-12ae-4322-8ff5-d30b2f36206c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:08.749764 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b6511a-12ae-4322-8ff5-d30b2f36206c service nova] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:08.750068 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b6511a-12ae-4322-8ff5-d30b2f36206c service nova] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:08.750343 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b6511a-12ae-4322-8ff5-d30b2f36206c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:08.765249 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 869/1702] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:45:08 2022] POST /identity/v3/projects => generated 371 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:45:08.768095 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80b6511a-12ae-4322-8ff5-d30b2f36206c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:08.769720 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80b6511a-12ae-4322-8ff5-d30b2f36206c service nova] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:08.774907 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a17de8ce-2700-4cf7-89bf-83b9897f2a46 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:08.810693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 834/1703] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:45:08 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 367 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:45:08.902718 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588dfcc8-b7ea-40c2-9c37-5b0166dabbe8 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:08.903003 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588dfcc8-b7ea-40c2-9c37-5b0166dabbe8 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:08.903283 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588dfcc8-b7ea-40c2-9c37-5b0166dabbe8 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:08.905038 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 835/1704] 10.222.0.25 () {54 vars in 912 bytes} [Fri Oct 28 09:45:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:45:08.923281 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43e4af5-5c06-4cf9-b7b3-9e35ddd557f0 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:08.923566 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43e4af5-5c06-4cf9-b7b3-9e35ddd557f0 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:08.923842 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43e4af5-5c06-4cf9-b7b3-9e35ddd557f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:08.957126 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:09.003204 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a17de8ce-2700-4cf7-89bf-83b9897f2a46 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:09.013528 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-b43e4af5-5c06-4cf9-b7b3-9e35ddd557f0 None None] MFA Rules not processed for user `1db6b98e87e64584932447c82fefc3b7`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:09.014819 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-b43e4af5-5c06-4cf9-b7b3-9e35ddd557f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:09.020884 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeda5f07-5bd4-477e-ad32-1da28a922f46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:09.021538 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeda5f07-5bd4-477e-ad32-1da28a922f46 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:09.021948 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeda5f07-5bd4-477e-ad32-1da28a922f46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:09.022324 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeda5f07-5bd4-477e-ad32-1da28a922f46 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:09.028681 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeda5f07-5bd4-477e-ad32-1da28a922f46 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:09.031858 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeda5f07-5bd4-477e-ad32-1da28a922f46 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:09.101826 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 870/1705] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:45:08 2022] POST /identity/v3/users => generated 286 bytes in 327 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:45:09.106698 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeda5f07-5bd4-477e-ad32-1da28a922f46 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:09.145461 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeda5f07-5bd4-477e-ad32-1da28a922f46 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:09.173850 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9982647d-fe38-4fa2-8583-5f80a8e0078f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:09.175019 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9982647d-fe38-4fa2-8583-5f80a8e0078f admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:09.175323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9982647d-fe38-4fa2-8583-5f80a8e0078f admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:09.175605 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9982647d-fe38-4fa2-8583-5f80a8e0078f admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:09.176333 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9982647d-fe38-4fa2-8583-5f80a8e0078f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:09.184775 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9982647d-fe38-4fa2-8583-5f80a8e0078f admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:09.199229 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 836/1706] 10.222.0.25 () {58 vars in 1009 bytes} [Fri Oct 28 09:45:08 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 277 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:45:09.214036 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 871/1707] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:45:09 2022] GET /identity/v3/roles => generated 1420 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:45:09.216545 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43e4af5-5c06-4cf9-b7b3-9e35ddd557f0 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:09.235709 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9982647d-fe38-4fa2-8583-5f80a8e0078f admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:09.239267 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43e4af5-5c06-4cf9-b7b3-9e35ddd557f0 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:09.242861 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-9982647d-fe38-4fa2-8583-5f80a8e0078f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:09.256081 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50064904-0e5e-4e68-8c03-cffedb069a19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:09.257323 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50064904-0e5e-4e68-8c03-cffedb069a19 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:09.257778 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50064904-0e5e-4e68-8c03-cffedb069a19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:09.258211 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50064904-0e5e-4e68-8c03-cffedb069a19 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:09.259333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50064904-0e5e-4e68-8c03-cffedb069a19 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:09.261342 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50064904-0e5e-4e68-8c03-cffedb069a19 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:09.269840 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9982647d-fe38-4fa2-8583-5f80a8e0078f admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:09.275590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 837/1708] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:45:09 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:45:09.279497 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50064904-0e5e-4e68-8c03-cffedb069a19 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:09.281082 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d96239f-14ba-45b6-b6d7-cfa6e1f47924 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:09.281716 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d96239f-14ba-45b6-b6d7-cfa6e1f47924 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:09.282006 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d96239f-14ba-45b6-b6d7-cfa6e1f47924 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:09.282286 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d96239f-14ba-45b6-b6d7-cfa6e1f47924 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:09.292288 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50064904-0e5e-4e68-8c03-cffedb069a19 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:09.297207 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8374ee9-4f73-4dcc-8070-38d0680844af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:09.297778 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8374ee9-4f73-4dcc-8070-38d0680844af admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:09.298060 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8374ee9-4f73-4dcc-8070-38d0680844af admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:09.298336 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8374ee9-4f73-4dcc-8070-38d0680844af admin admin] PATH_INFO: `/v3/projects/7332b23691264d34b110a6a5955c8ba1/users/5e69e5fa2fe742899f96277df3697a79/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:09.314864 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d96239f-14ba-45b6-b6d7-cfa6e1f47924 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:09.316244 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d96239f-14ba-45b6-b6d7-cfa6e1f47924 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:09.344458 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8374ee9-4f73-4dcc-8070-38d0680844af admin admin] RBAC: Authorizing `identity:create_grant(project_id=7332b23691264d34b110a6a5955c8ba1, user_id=5e69e5fa2fe742899f96277df3697a79, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:09.347838 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8374ee9-4f73-4dcc-8070-38d0680844af admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:09.354306 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 872/1709] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:45:09 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:45:09.382920 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 838/1710] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:45:09 2022] PUT /identity/v3/projects/7332b23691264d34b110a6a5955c8ba1/users/5e69e5fa2fe742899f96277df3697a79/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:45:09.394753 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd8d9a2-0dc2-474b-a3f9-0addc994cb90 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:09.395229 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd8d9a2-0dc2-474b-a3f9-0addc994cb90 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:09.395647 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd8d9a2-0dc2-474b-a3f9-0addc994cb90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:09.450903 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-5fd8d9a2-0dc2-474b-a3f9-0addc994cb90 None None] MFA Rules not processed for user `5e69e5fa2fe742899f96277df3697a79`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:09.452873 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-5fd8d9a2-0dc2-474b-a3f9-0addc994cb90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:09.543421 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:09.578032 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 873/1711] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:09 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:45:09.579284 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8374ee9-4f73-4dcc-8070-38d0680844af admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:09.724396 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8374ee9-4f73-4dcc-8070-38d0680844af admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:09.735225 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7ba3a6-a17d-4499-90cc-86a31cd3e609 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:09.736048 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7ba3a6-a17d-4499-90cc-86a31cd3e609 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:09.736445 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7ba3a6-a17d-4499-90cc-86a31cd3e609 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:09.736800 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7ba3a6-a17d-4499-90cc-86a31cd3e609 admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:09.738111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a7ba3a6-a17d-4499-90cc-86a31cd3e609 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:09.740366 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a7ba3a6-a17d-4499-90cc-86a31cd3e609 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:09.784790 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 839/1712] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:45:09 2022] POST /identity/v3/projects => generated 371 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:45:09.788877 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd8d9a2-0dc2-474b-a3f9-0addc994cb90 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:09.808091 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd8d9a2-0dc2-474b-a3f9-0addc994cb90 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:09.955139 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df8b04c4-bce6-4f97-a86d-4fc8c04dd2d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:09.955779 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8b04c4-bce6-4f97-a86d-4fc8c04dd2d8 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:09.956062 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8b04c4-bce6-4f97-a86d-4fc8c04dd2d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:09.956428 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8b04c4-bce6-4f97-a86d-4fc8c04dd2d8 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:09.957093 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df8b04c4-bce6-4f97-a86d-4fc8c04dd2d8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:09.959795 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df8b04c4-bce6-4f97-a86d-4fc8c04dd2d8 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:10.015809 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7ba3a6-a17d-4499-90cc-86a31cd3e609 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:10.021382 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 874/1713] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:45:09 2022] POST /identity/v3/users => generated 292 bytes in 229 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:45:10.027648 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7ba3a6-a17d-4499-90cc-86a31cd3e609 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:10.041198 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6491426d-b17e-4591-aaa0-9841e6e8fb28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:10.041896 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6491426d-b17e-4591-aaa0-9841e6e8fb28 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:10.042361 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6491426d-b17e-4591-aaa0-9841e6e8fb28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:10.042821 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6491426d-b17e-4591-aaa0-9841e6e8fb28 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:10.043683 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6491426d-b17e-4591-aaa0-9841e6e8fb28 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:10.047065 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6491426d-b17e-4591-aaa0-9841e6e8fb28 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:10.080882 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df8b04c4-bce6-4f97-a86d-4fc8c04dd2d8 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:10.099704 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df8b04c4-bce6-4f97-a86d-4fc8c04dd2d8 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:10.100994 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 840/1714] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:45:10 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:45:10.116931 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-398845ee-0895-4cd8-a926-52f24cd74cab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:10.118038 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398845ee-0895-4cd8-a926-52f24cd74cab admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:10.118864 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398845ee-0895-4cd8-a926-52f24cd74cab admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:10.119373 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398845ee-0895-4cd8-a926-52f24cd74cab admin admin] PATH_INFO: `/v3/projects/ee3e0a7af3f44970a619bf123fdf2465/users/0fa198d1bbe04387b2d045f741c08777/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:10.180442 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-398845ee-0895-4cd8-a926-52f24cd74cab admin admin] RBAC: Authorizing `identity:create_grant(project_id=ee3e0a7af3f44970a619bf123fdf2465, user_id=0fa198d1bbe04387b2d045f741c08777, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:10.186543 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-398845ee-0895-4cd8-a926-52f24cd74cab admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:10.226180 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 875/1715] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:45:10 2022] PUT /identity/v3/projects/ee3e0a7af3f44970a619bf123fdf2465/users/0fa198d1bbe04387b2d045f741c08777/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:45:10.230784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6491426d-b17e-4591-aaa0-9841e6e8fb28 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:10.245814 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6491426d-b17e-4591-aaa0-9841e6e8fb28 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:10.353390 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:10.387506 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21fcbe81-5a7e-41a1-ac1e-806fb0f33cb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:10.391968 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fcbe81-5a7e-41a1-ac1e-806fb0f33cb1 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:10.396392 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fcbe81-5a7e-41a1-ac1e-806fb0f33cb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:10.398069 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21fcbe81-5a7e-41a1-ac1e-806fb0f33cb1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:10.398816 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21fcbe81-5a7e-41a1-ac1e-806fb0f33cb1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:10.404815 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21fcbe81-5a7e-41a1-ac1e-806fb0f33cb1 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:10.438347 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 841/1716] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:45:10 2022] GET /identity/v3/roles => generated 1420 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:45:10.442653 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-398845ee-0895-4cd8-a926-52f24cd74cab admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:10.459741 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-398845ee-0895-4cd8-a926-52f24cd74cab admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:10.473774 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff869b4-2fb5-4e5f-85c8-84f8f3f69035 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:10.474926 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff869b4-2fb5-4e5f-85c8-84f8f3f69035 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:10.476081 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff869b4-2fb5-4e5f-85c8-84f8f3f69035 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:10.476581 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff869b4-2fb5-4e5f-85c8-84f8f3f69035 admin admin] PATH_INFO: `/v3/projects/ee3e0a7af3f44970a619bf123fdf2465/users/0fa198d1bbe04387b2d045f741c08777/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:10.482095 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff869b4-2fb5-4e5f-85c8-84f8f3f69035 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ee3e0a7af3f44970a619bf123fdf2465, user_id=0fa198d1bbe04387b2d045f741c08777, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:10.485763 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff869b4-2fb5-4e5f-85c8-84f8f3f69035 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:10.525095 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 876/1717] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:45:10 2022] PUT /identity/v3/projects/ee3e0a7af3f44970a619bf123fdf2465/users/0fa198d1bbe04387b2d045f741c08777/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 83 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:45:10.535377 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73a9263-fdc1-4811-b7f9-67bf54d19b4c None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:10.536593 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73a9263-fdc1-4811-b7f9-67bf54d19b4c None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:10.537032 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73a9263-fdc1-4811-b7f9-67bf54d19b4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:10.604753 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-f73a9263-fdc1-4811-b7f9-67bf54d19b4c None None] MFA Rules not processed for user `0fa198d1bbe04387b2d045f741c08777`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:10.610785 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-f73a9263-fdc1-4811-b7f9-67bf54d19b4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:10.733527 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff869b4-2fb5-4e5f-85c8-84f8f3f69035 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:10.736052 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-9ff869b4-2fb5-4e5f-85c8-84f8f3f69035 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:10.814437 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 842/1718] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:10 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 280 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:45:10.826716 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73a9263-fdc1-4811-b7f9-67bf54d19b4c None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:10.828282 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-f73a9263-fdc1-4811-b7f9-67bf54d19b4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:10.893023 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff869b4-2fb5-4e5f-85c8-84f8f3f69035 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:10.901538 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449e5a6a-24d0-475c-bf22-8b7ce863b2cc service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3f1d95049d2d4593bf52670b1a0035fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:10.901538 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449e5a6a-24d0-475c-bf22-8b7ce863b2cc service placement] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:10.901538 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449e5a6a-24d0-475c-bf22-8b7ce863b2cc service placement] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:10.901538 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449e5a6a-24d0-475c-bf22-8b7ce863b2cc service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:10.905241 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-449e5a6a-24d0-475c-bf22-8b7ce863b2cc service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:10.928114 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449e5a6a-24d0-475c-bf22-8b7ce863b2cc service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:10.929880 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449e5a6a-24d0-475c-bf22-8b7ce863b2cc service placement] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:10.959502 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 877/1719] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:45:10 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 226 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:45:10.965739 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73a9263-fdc1-4811-b7f9-67bf54d19b4c None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:10.970811 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83961ea8-7236-4f28-b844-6f04013c891c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:10.971839 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83961ea8-7236-4f28-b844-6f04013c891c service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:10.972430 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83961ea8-7236-4f28-b844-6f04013c891c service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:10.973063 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83961ea8-7236-4f28-b844-6f04013c891c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:10.993551 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83961ea8-7236-4f28-b844-6f04013c891c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:10.996808 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83961ea8-7236-4f28-b844-6f04013c891c service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:11.012234 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 843/1720] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:45:10 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 186 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:45:11.054689 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449e5a6a-24d0-475c-bf22-8b7ce863b2cc service placement] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:11.056136 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-449e5a6a-24d0-475c-bf22-8b7ce863b2cc service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:11.097302 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449e5a6a-24d0-475c-bf22-8b7ce863b2cc service placement] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:11.128696 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931deded-55ba-4ab0-874d-1e6379b46782 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:11.129314 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931deded-55ba-4ab0-874d-1e6379b46782 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:11.129781 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-931deded-55ba-4ab0-874d-1e6379b46782 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:11.140634 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46294e5b-f2ba-41c6-9da7-80cb84bb9f96 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:11.147681 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46294e5b-f2ba-41c6-9da7-80cb84bb9f96 service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:11.147985 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46294e5b-f2ba-41c6-9da7-80cb84bb9f96 service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:11.148265 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46294e5b-f2ba-41c6-9da7-80cb84bb9f96 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:11.156468 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-46294e5b-f2ba-41c6-9da7-80cb84bb9f96 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:11.176502 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-931deded-55ba-4ab0-874d-1e6379b46782 None None] MFA Rules not processed for user `ed9a4c24cbfb4689b7f984219c7503a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:11.179145 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-931deded-55ba-4ab0-874d-1e6379b46782 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:11.272790 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:11.333752 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 844/1721] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:11 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:45:11.333752 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-931deded-55ba-4ab0-874d-1e6379b46782 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:11.386307 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-931deded-55ba-4ab0-874d-1e6379b46782 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:11.387542 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46294e5b-f2ba-41c6-9da7-80cb84bb9f96 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:11.389149 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46294e5b-f2ba-41c6-9da7-80cb84bb9f96 service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:11.403098 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd488970-dbe7-46cf-9dc8-19fa8ca9dd2b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:11.405631 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd488970-dbe7-46cf-9dc8-19fa8ca9dd2b service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:11.408907 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd488970-dbe7-46cf-9dc8-19fa8ca9dd2b service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:11.410385 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd488970-dbe7-46cf-9dc8-19fa8ca9dd2b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:11.440898 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 878/1722] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:45:11 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 387 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:45:11.470799 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:11.650069 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd488970-dbe7-46cf-9dc8-19fa8ca9dd2b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:11.656800 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd488970-dbe7-46cf-9dc8-19fa8ca9dd2b service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:11.718177 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 845/1723] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:45:11 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 387 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:45:13.877304 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2758808b-6f7b-473a-bcd2-e8832869434a None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:13.877973 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2758808b-6f7b-473a-bcd2-e8832869434a None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:13.877973 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2758808b-6f7b-473a-bcd2-e8832869434a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:13.939843 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-2758808b-6f7b-473a-bcd2-e8832869434a None None] MFA Rules not processed for user `715a782e5a56431e88e4fe5a3b0a62aa`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:13.941437 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-2758808b-6f7b-473a-bcd2-e8832869434a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:14.064790 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 879/1724] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:13 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:45:14.335642 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aea41a7-bb44-4abc-b1ab-394ac3394f58 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:14.336175 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aea41a7-bb44-4abc-b1ab-394ac3394f58 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:14.336269 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aea41a7-bb44-4abc-b1ab-394ac3394f58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:14.419543 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-5aea41a7-bb44-4abc-b1ab-394ac3394f58 None None] MFA Rules not processed for user `e65d298cca0f478eba502edf5bd13954`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:14.420802 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-5aea41a7-bb44-4abc-b1ab-394ac3394f58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:14.437334 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3228d05c-afdf-4dd7-87ab-44975eb9ae96 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:14.437631 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3228d05c-afdf-4dd7-87ab-44975eb9ae96 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:14.437915 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3228d05c-afdf-4dd7-87ab-44975eb9ae96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:14.562684 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-3228d05c-afdf-4dd7-87ab-44975eb9ae96 None None] MFA Rules not processed for user `e451d264e4964409b09e0b96ae8b983e`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:14.563953 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-3228d05c-afdf-4dd7-87ab-44975eb9ae96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:14.686246 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 846/1725] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:14 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 347 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:45:14.696828 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aea41a7-bb44-4abc-b1ab-394ac3394f58 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:14.772935 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 880/1726] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:14 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 336 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:45:14.785272 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09d7733-ef3a-43e1-ad5c-89e38453eeb6 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:14.785541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09d7733-ef3a-43e1-ad5c-89e38453eeb6 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:14.785816 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09d7733-ef3a-43e1-ad5c-89e38453eeb6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:14.838635 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-c09d7733-ef3a-43e1-ad5c-89e38453eeb6 None None] MFA Rules not processed for user `e451d264e4964409b09e0b96ae8b983e`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:14.846081 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-c09d7733-ef3a-43e1-ad5c-89e38453eeb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:14.866510 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 881/1727] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:14 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:45:14.907999 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aea41a7-bb44-4abc-b1ab-394ac3394f58 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:14.940208 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9d3403-8f4d-4628-95dc-345f701d8986 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:14.940208 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9d3403-8f4d-4628-95dc-345f701d8986 service nova] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:14.940208 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9d3403-8f4d-4628-95dc-345f701d8986 service nova] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:14.940208 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9d3403-8f4d-4628-95dc-345f701d8986 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:14.965797 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9d3403-8f4d-4628-95dc-345f701d8986 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:14.967708 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9d3403-8f4d-4628-95dc-345f701d8986 service nova] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:14.997419 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 847/1728] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:45:14 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:45:15.139340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c09d7733-ef3a-43e1-ad5c-89e38453eeb6 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:15.154938 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b29cd10c-51ff-4dba-9119-99d08bb549ad None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:15.155222 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b29cd10c-51ff-4dba-9119-99d08bb549ad None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:15.155499 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b29cd10c-51ff-4dba-9119-99d08bb549ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:15.194699 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c09d7733-ef3a-43e1-ad5c-89e38453eeb6 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:15.215239 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f8b556-a7dd-48a4-8249-222258ab0669 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:15.216648 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f8b556-a7dd-48a4-8249-222258ab0669 service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:15.217031 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f8b556-a7dd-48a4-8249-222258ab0669 service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:15.218074 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f8b556-a7dd-48a4-8249-222258ab0669 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:15.232094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f8b556-a7dd-48a4-8249-222258ab0669 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:15.234818 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f8b556-a7dd-48a4-8249-222258ab0669 service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:15.246897 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-b29cd10c-51ff-4dba-9119-99d08bb549ad None None] MFA Rules not processed for user `a4237ca76acd4f7483aaf66e1f5d2eea`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:15.251458 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-b29cd10c-51ff-4dba-9119-99d08bb549ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:15.255679 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 882/1729] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:45:15 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:45:15.289888 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:15.468296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 848/1730] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:15 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 314 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:45:16.422907 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c81dc9-0837-421c-a43b-06329274a198 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:16.423927 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c81dc9-0837-421c-a43b-06329274a198 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:16.424482 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c81dc9-0837-421c-a43b-06329274a198 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:16.426359 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 883/1731] 10.222.0.25 () {54 vars in 912 bytes} [Fri Oct 28 09:45:16 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:45:16.438707 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3c276e-7ceb-48b7-8f80-b14a31cb35e2 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:16.439206 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3c276e-7ceb-48b7-8f80-b14a31cb35e2 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:16.439702 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3c276e-7ceb-48b7-8f80-b14a31cb35e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:16.484799 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c81dc9-0837-421c-a43b-06329274a198 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:16.486540 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-54c81dc9-0837-421c-a43b-06329274a198 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:16.536241 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c81dc9-0837-421c-a43b-06329274a198 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:16.538237 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-fe3c276e-7ceb-48b7-8f80-b14a31cb35e2 None None] MFA Rules not processed for user `1db6b98e87e64584932447c82fefc3b7`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:16.544669 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-fe3c276e-7ceb-48b7-8f80-b14a31cb35e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:16.564774 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ece3380-6bda-4387-b21c-5c1158003c2a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:16.565682 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ece3380-6bda-4387-b21c-5c1158003c2a service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:16.566134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ece3380-6bda-4387-b21c-5c1158003c2a service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:16.595355 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ece3380-6bda-4387-b21c-5c1158003c2a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:16.638102 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ece3380-6bda-4387-b21c-5c1158003c2a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:16.652736 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ece3380-6bda-4387-b21c-5c1158003c2a service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:16.675847 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 849/1732] 10.222.0.25 () {58 vars in 1009 bytes} [Fri Oct 28 09:45:16 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 238 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:45:16.695561 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe3c276e-7ceb-48b7-8f80-b14a31cb35e2 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:16.716586 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe3c276e-7ceb-48b7-8f80-b14a31cb35e2 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:16.722787 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e737d56-1054-4145-8c92-c74f42999105 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:16.723274 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 884/1733] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:45:16 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 239 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:45:16.725089 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e737d56-1054-4145-8c92-c74f42999105 service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:16.725451 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e737d56-1054-4145-8c92-c74f42999105 service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:16.725796 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e737d56-1054-4145-8c92-c74f42999105 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:16.742474 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e737d56-1054-4145-8c92-c74f42999105 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:16.747849 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e737d56-1054-4145-8c92-c74f42999105 service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:16.770608 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 850/1734] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:45:16 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 76 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:45:19.177643 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac49a82f-178c-46f6-9c7c-64f2a48507cd None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:19.180375 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac49a82f-178c-46f6-9c7c-64f2a48507cd None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:19.180956 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac49a82f-178c-46f6-9c7c-64f2a48507cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:19.251624 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-ac49a82f-178c-46f6-9c7c-64f2a48507cd None None] MFA Rules not processed for user `e65d298cca0f478eba502edf5bd13954`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:19.254249 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ac49a82f-178c-46f6-9c7c-64f2a48507cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:19.289162 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 885/1735] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:19 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:45:20.279241 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9e787e-e355-4efc-8e9b-f3e71c9c4237 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:20.280078 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9e787e-e355-4efc-8e9b-f3e71c9c4237 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:20.280628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9e787e-e355-4efc-8e9b-f3e71c9c4237 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:20.319289 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-fe9e787e-e355-4efc-8e9b-f3e71c9c4237 None None] MFA Rules not processed for user `a4237ca76acd4f7483aaf66e1f5d2eea`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:20.320511 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-fe9e787e-e355-4efc-8e9b-f3e71c9c4237 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:20.340401 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 851/1736] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:20 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 62 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:45:20.884162 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e248b358-7d53-495a-961a-e2a5bdfd6e11 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:20.884616 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e248b358-7d53-495a-961a-e2a5bdfd6e11 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:20.884724 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e248b358-7d53-495a-961a-e2a5bdfd6e11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:20.987964 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:21.086840 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-e248b358-7d53-495a-961a-e2a5bdfd6e11 None None] MFA Rules not processed for user `97c8877c74194a6bbbf25b3c8f5b7dbd`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:21.088125 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e248b358-7d53-495a-961a-e2a5bdfd6e11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:21.295089 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 886/1737] 10.222.0.25 () {58 vars in 1009 bytes} [Fri Oct 28 09:45:20 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 412 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:45:21.655519 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9e787e-e355-4efc-8e9b-f3e71c9c4237 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:21.657133 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-fe9e787e-e355-4efc-8e9b-f3e71c9c4237 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:21.933278 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9e787e-e355-4efc-8e9b-f3e71c9c4237 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:21.946162 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f740d5a-8219-4bab-9db9-3dbe0a0cb0a4 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c6cffe0d56e45c4b8fb54fbe5df6094', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:21.947454 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f740d5a-8219-4bab-9db9-3dbe0a0cb0a4 service swift] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:21.947881 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f740d5a-8219-4bab-9db9-3dbe0a0cb0a4 service swift] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:21.948174 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f740d5a-8219-4bab-9db9-3dbe0a0cb0a4 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:21.977633 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f740d5a-8219-4bab-9db9-3dbe0a0cb0a4 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:21.979288 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f740d5a-8219-4bab-9db9-3dbe0a0cb0a4 service swift] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:21.994681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 852/1738] 10.222.0.25 () {60 vars in 1384 bytes} [Fri Oct 28 09:45:21 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 340 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 28 09:45:24.438876 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4603cb8c-f3e7-44d6-a07e-c5d75c16b218 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:24.439382 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4603cb8c-f3e7-44d6-a07e-c5d75c16b218 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:24.439382 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4603cb8c-f3e7-44d6-a07e-c5d75c16b218 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:24.515026 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-4603cb8c-f3e7-44d6-a07e-c5d75c16b218 None None] MFA Rules not processed for user `1db6b98e87e64584932447c82fefc3b7`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:24.516814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4603cb8c-f3e7-44d6-a07e-c5d75c16b218 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:24.591130 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 887/1739] 10.222.0.55 () {58 vars in 1015 bytes} [Fri Oct 28 09:45:24 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:45:25.539108 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f740d5a-8219-4bab-9db9-3dbe0a0cb0a4 service swift] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:25.563018 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f740d5a-8219-4bab-9db9-3dbe0a0cb0a4 service swift] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:25.606368 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd70176-c664-42a1-a988-ce7a148e0b0f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:25.608984 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd70176-c664-42a1-a988-ce7a148e0b0f service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:25.610674 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd70176-c664-42a1-a988-ce7a148e0b0f service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:25.611238 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd70176-c664-42a1-a988-ce7a148e0b0f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:25.643236 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd70176-c664-42a1-a988-ce7a148e0b0f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:25.646612 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd70176-c664-42a1-a988-ce7a148e0b0f service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:25.688192 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 853/1740] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:45:25 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:45:28.784651 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a1a981-cb0d-4082-90ec-713856e2ed75 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:28.785256 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a1a981-cb0d-4082-90ec-713856e2ed75 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:28.785256 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a1a981-cb0d-4082-90ec-713856e2ed75 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:28.786622 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 888/1741] 10.222.0.25 () {54 vars in 921 bytes} [Fri Oct 28 09:45:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 09:45:28.797947 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59136bd5-5d28-4e0c-bbcd-1d87ef4e7a1e None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:28.798870 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59136bd5-5d28-4e0c-bbcd-1d87ef4e7a1e None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:28.799303 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59136bd5-5d28-4e0c-bbcd-1d87ef4e7a1e None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:28.801062 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 854/1742] 10.222.0.25 () {54 vars in 921 bytes} [Fri Oct 28 09:45:28 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:45:28.810146 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f3096c-a793-4d2d-8c5b-668175abba9f None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:28.810729 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f3096c-a793-4d2d-8c5b-668175abba9f None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:28.812282 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f3096c-a793-4d2d-8c5b-668175abba9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:28.818525 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722714fb-6af3-4452-9a48-ded70eff8a10 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:28.819421 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722714fb-6af3-4452-9a48-ded70eff8a10 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:28.820402 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722714fb-6af3-4452-9a48-ded70eff8a10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:28.848148 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:28.852900 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-69f3096c-a793-4d2d-8c5b-668175abba9f None None] MFA Rules not processed for user `688ceb201d304c2da5373c3cf36c2d79`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:28.854859 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-69f3096c-a793-4d2d-8c5b-668175abba9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:28.866325 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-722714fb-6af3-4452-9a48-ded70eff8a10 None None] MFA Rules not processed for user `688ceb201d304c2da5373c3cf36c2d79`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:28.867625 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-722714fb-6af3-4452-9a48-ded70eff8a10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:28.868367 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 889/1743] 10.222.0.25 () {58 vars in 1018 bytes} [Fri Oct 28 09:45:28 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 61 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:45:28.877849 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 855/1744] 10.222.0.25 () {58 vars in 1018 bytes} [Fri Oct 28 09:45:28 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 63 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:45:28.880759 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69f3096c-a793-4d2d-8c5b-668175abba9f None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:28.890172 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722714fb-6af3-4452-9a48-ded70eff8a10 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:28.896569 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69f3096c-a793-4d2d-8c5b-668175abba9f None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:28.905580 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722714fb-6af3-4452-9a48-ded70eff8a10 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:28.908208 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aafd1a6a-4e92-43dd-bc08-c8855d5452ca service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:28.908841 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafd1a6a-4e92-43dd-bc08-c8855d5452ca service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:28.909124 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafd1a6a-4e92-43dd-bc08-c8855d5452ca service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:28.909402 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafd1a6a-4e92-43dd-bc08-c8855d5452ca service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:28.911700 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1cd655-b1a9-452a-9758-7e1087361baf service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:28.912394 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1cd655-b1a9-452a-9758-7e1087361baf service nova] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:28.913274 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1cd655-b1a9-452a-9758-7e1087361baf service nova] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:28.913730 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1cd655-b1a9-452a-9758-7e1087361baf service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:28.925411 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aafd1a6a-4e92-43dd-bc08-c8855d5452ca service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:28.927079 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aafd1a6a-4e92-43dd-bc08-c8855d5452ca service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:28.927539 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b1cd655-b1a9-452a-9758-7e1087361baf service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:28.929379 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b1cd655-b1a9-452a-9758-7e1087361baf service nova] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:28.944527 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 856/1745] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:45:28 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 55 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:45:28.959713 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 890/1746] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:45:28 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 80 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:45:38.486718 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1cd655-b1a9-452a-9758-7e1087361baf service nova] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:38.489219 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-6b1cd655-b1a9-452a-9758-7e1087361baf service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:38.592275 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1cd655-b1a9-452a-9758-7e1087361baf service nova] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:38.617416 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af384d15-3081-4663-a697-5f0e684e44a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:38.618740 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af384d15-3081-4663-a697-5f0e684e44a0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:38.619209 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af384d15-3081-4663-a697-5f0e684e44a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:38.619706 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af384d15-3081-4663-a697-5f0e684e44a0 admin admin] PATH_INFO: `/v3/users/5a4b6abb24e4425f9427d88487ad2062` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:38.621342 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af384d15-3081-4663-a697-5f0e684e44a0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5a4b6abb24e4425f9427d88487ad2062)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:38.624069 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af384d15-3081-4663-a697-5f0e684e44a0 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:38.761150 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-af384d15-3081-4663-a697-5f0e684e44a0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5a4b6abb24e4425f9427d88487ad2062'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:38.776178 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-af384d15-3081-4663-a697-5f0e684e44a0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5a4b6abb24e4425f9427d88487ad2062'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:38.776972 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-af384d15-3081-4663-a697-5f0e684e44a0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5a4b6abb24e4425f9427d88487ad2062'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:38.777609 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-af384d15-3081-4663-a697-5f0e684e44a0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5a4b6abb24e4425f9427d88487ad2062'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:38.795326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-af384d15-3081-4663-a697-5f0e684e44a0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5a4b6abb24e4425f9427d88487ad2062'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:38.836910 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 857/1747] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:45:38 2022] DELETE /identity/v3/users/5a4b6abb24e4425f9427d88487ad2062 => generated 0 bytes in 351 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:45:38.841357 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d31159-8d98-422a-b18e-f80432bb9508 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:38.841918 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d31159-8d98-422a-b18e-f80432bb9508 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:38.841918 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d31159-8d98-422a-b18e-f80432bb9508 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:38.861877 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-31d31159-8d98-422a-b18e-f80432bb9508 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:38.863756 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-31d31159-8d98-422a-b18e-f80432bb9508 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:38.991752 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 891/1748] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:45:39.014751 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af384d15-3081-4663-a697-5f0e684e44a0 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:39.023784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-af384d15-3081-4663-a697-5f0e684e44a0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:39.139962 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af384d15-3081-4663-a697-5f0e684e44a0 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:39.149960 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc08b98-e5d4-4e42-bcb4-90bea1b9a14c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:39.151633 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc08b98-e5d4-4e42-bcb4-90bea1b9a14c service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:39.152316 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc08b98-e5d4-4e42-bcb4-90bea1b9a14c service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:39.153115 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc08b98-e5d4-4e42-bcb4-90bea1b9a14c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:39.169205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bc08b98-e5d4-4e42-bcb4-90bea1b9a14c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:39.176972 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bc08b98-e5d4-4e42-bcb4-90bea1b9a14c service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:39.188097 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 858/1749] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:45:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:45:40.443511 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d31159-8d98-422a-b18e-f80432bb9508 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:40.445039 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-31d31159-8d98-422a-b18e-f80432bb9508 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:40.466729 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d31159-8d98-422a-b18e-f80432bb9508 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:40.472984 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6663d925-3e95-4026-a33f-226de1e34fb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:40.474147 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6663d925-3e95-4026-a33f-226de1e34fb9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:40.474863 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6663d925-3e95-4026-a33f-226de1e34fb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:40.484405 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6663d925-3e95-4026-a33f-226de1e34fb9 admin admin] PATH_INFO: `/v3/projects/d8700031698d4e7eb69fb34d2d1c73e8` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:40.486306 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6663d925-3e95-4026-a33f-226de1e34fb9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d8700031698d4e7eb69fb34d2d1c73e8)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:40.490361 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6663d925-3e95-4026-a33f-226de1e34fb9 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:40.563615 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-6663d925-3e95-4026-a33f-226de1e34fb9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd8700031698d4e7eb69fb34d2d1c73e8', 'request_id': 'req-6663d925-3e95-4026-a33f-226de1e34fb9'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:40.572165 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-6663d925-3e95-4026-a33f-226de1e34fb9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd8700031698d4e7eb69fb34d2d1c73e8', 'request_id': 'req-6663d925-3e95-4026-a33f-226de1e34fb9'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:40.581383 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:40.588130 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-6663d925-3e95-4026-a33f-226de1e34fb9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd8700031698d4e7eb69fb34d2d1c73e8', 'request_id': 'req-6663d925-3e95-4026-a33f-226de1e34fb9'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:40.601846 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-6663d925-3e95-4026-a33f-226de1e34fb9 admin admin] The token cache is being invalidate because project d8700031698d4e7eb69fb34d2d1c73e8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:45:40.602522 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-6663d925-3e95-4026-a33f-226de1e34fb9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:40.603371 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-6663d925-3e95-4026-a33f-226de1e34fb9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:40.604916 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 892/1750] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:45:40 2022] DELETE /identity/v3/projects/d8700031698d4e7eb69fb34d2d1c73e8 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:45:40.609619 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc08b98-e5d4-4e42-bcb4-90bea1b9a14c service neutron] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:40.620286 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc08b98-e5d4-4e42-bcb4-90bea1b9a14c service neutron] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:40.624397 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:40.686127 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a375f8-3a87-42fb-9f2d-9c398b36afa2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:40.687112 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a375f8-3a87-42fb-9f2d-9c398b36afa2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:40.687548 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a375f8-3a87-42fb-9f2d-9c398b36afa2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:40.687659 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a375f8-3a87-42fb-9f2d-9c398b36afa2 admin admin] PATH_INFO: `/v3/users/b3838c4e1dfe40cdbcdc28964810eb4b` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:40.689134 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a375f8-3a87-42fb-9f2d-9c398b36afa2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b3838c4e1dfe40cdbcdc28964810eb4b)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:40.691614 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a375f8-3a87-42fb-9f2d-9c398b36afa2 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:40.742667 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-53a375f8-3a87-42fb-9f2d-9c398b36afa2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b3838c4e1dfe40cdbcdc28964810eb4b'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:40.750610 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-53a375f8-3a87-42fb-9f2d-9c398b36afa2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b3838c4e1dfe40cdbcdc28964810eb4b'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:40.751234 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-53a375f8-3a87-42fb-9f2d-9c398b36afa2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b3838c4e1dfe40cdbcdc28964810eb4b'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:40.751847 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-53a375f8-3a87-42fb-9f2d-9c398b36afa2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b3838c4e1dfe40cdbcdc28964810eb4b'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:40.761140 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-53a375f8-3a87-42fb-9f2d-9c398b36afa2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b3838c4e1dfe40cdbcdc28964810eb4b'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:40.778658 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 859/1751] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:45:40 2022] DELETE /identity/v3/users/b3838c4e1dfe40cdbcdc28964810eb4b => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:45:41.098360 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6663d925-3e95-4026-a33f-226de1e34fb9 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:41.101534 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-6663d925-3e95-4026-a33f-226de1e34fb9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:41.124553 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6663d925-3e95-4026-a33f-226de1e34fb9 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:41.240890 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa44137f-0d21-4be7-b9bc-e9edb64956ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:41.241845 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa44137f-0d21-4be7-b9bc-e9edb64956ad admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:41.242273 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa44137f-0d21-4be7-b9bc-e9edb64956ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:41.242696 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa44137f-0d21-4be7-b9bc-e9edb64956ad admin admin] PATH_INFO: `/v3/projects/ba157963b89d4f0c928fa9bc50792d1d` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:41.244171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa44137f-0d21-4be7-b9bc-e9edb64956ad admin admin] RBAC: Authorizing `identity:delete_project(project_id=ba157963b89d4f0c928fa9bc50792d1d)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:41.246413 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa44137f-0d21-4be7-b9bc-e9edb64956ad admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:41.322147 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-fa44137f-0d21-4be7-b9bc-e9edb64956ad admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ba157963b89d4f0c928fa9bc50792d1d', 'request_id': 'req-fa44137f-0d21-4be7-b9bc-e9edb64956ad'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:41.334211 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-fa44137f-0d21-4be7-b9bc-e9edb64956ad admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ba157963b89d4f0c928fa9bc50792d1d', 'request_id': 'req-fa44137f-0d21-4be7-b9bc-e9edb64956ad'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:41.347452 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-fa44137f-0d21-4be7-b9bc-e9edb64956ad admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ba157963b89d4f0c928fa9bc50792d1d', 'request_id': 'req-fa44137f-0d21-4be7-b9bc-e9edb64956ad'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:41.364192 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-fa44137f-0d21-4be7-b9bc-e9edb64956ad admin admin] The token cache is being invalidate because project ba157963b89d4f0c928fa9bc50792d1d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:45:41.364482 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-fa44137f-0d21-4be7-b9bc-e9edb64956ad admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:41.365242 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-fa44137f-0d21-4be7-b9bc-e9edb64956ad admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:41.367224 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 893/1752] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:45:41 2022] DELETE /identity/v3/projects/ba157963b89d4f0c928fa9bc50792d1d => generated 0 bytes in 270 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:45:41.373755 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53be3e6b-6767-401f-9901-a9efab849c5a None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:41.373874 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53be3e6b-6767-401f-9901-a9efab849c5a None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:41.374414 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53be3e6b-6767-401f-9901-a9efab849c5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:41.411408 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-53be3e6b-6767-401f-9901-a9efab849c5a None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:41.415502 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-53be3e6b-6767-401f-9901-a9efab849c5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:41.435368 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa44137f-0d21-4be7-b9bc-e9edb64956ad admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:41.437546 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-fa44137f-0d21-4be7-b9bc-e9edb64956ad admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:41.500998 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 860/1753] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:45:41.509280 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c072e85-343f-4139-a372-d1b2000df576 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:41.509956 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c072e85-343f-4139-a372-d1b2000df576 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:41.510458 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c072e85-343f-4139-a372-d1b2000df576 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:41.532701 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-7c072e85-343f-4139-a372-d1b2000df576 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:41.534557 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-7c072e85-343f-4139-a372-d1b2000df576 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:41.550237 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 861/1754] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:45:41.563304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c072e85-343f-4139-a372-d1b2000df576 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:41.592528 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c072e85-343f-4139-a372-d1b2000df576 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:41.607770 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa44137f-0d21-4be7-b9bc-e9edb64956ad admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:41.608314 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d4397b-5d6c-4871-a4b5-0f37e99145e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:41.608569 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d4397b-5d6c-4871-a4b5-0f37e99145e8 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:41.608883 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d4397b-5d6c-4871-a4b5-0f37e99145e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:41.609919 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d4397b-5d6c-4871-a4b5-0f37e99145e8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:41.610699 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d4397b-5d6c-4871-a4b5-0f37e99145e8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:45:41.611008 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d4397b-5d6c-4871-a4b5-0f37e99145e8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:41.617572 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de86e839-9da2-4db6-adad-abea8f9db7e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:41.618606 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d4397b-5d6c-4871-a4b5-0f37e99145e8 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:41.619659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de86e839-9da2-4db6-adad-abea8f9db7e4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:41.620056 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de86e839-9da2-4db6-adad-abea8f9db7e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:41.620584 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de86e839-9da2-4db6-adad-abea8f9db7e4 admin admin] PATH_INFO: `/v3/users/39e891d5d8a6431fb6f5343c543214c5` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:41.629931 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de86e839-9da2-4db6-adad-abea8f9db7e4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=39e891d5d8a6431fb6f5343c543214c5)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:41.629931 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: WARNING py.warnings [None req-de86e839-9da2-4db6-adad-abea8f9db7e4 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Oct 28 09:45:41.629931 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: warnings.warn(msg) Oct 28 09:45:41.629931 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: Oct 28 09:45:41.629931 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de86e839-9da2-4db6-adad-abea8f9db7e4 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:41.671514 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 862/1755] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:45:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 112 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:45:41.676015 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d4397b-5d6c-4871-a4b5-0f37e99145e8 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:41.680544 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:41.708638 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d4397b-5d6c-4871-a4b5-0f37e99145e8 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:41.720545 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c47c05e3-aec1-413c-92eb-4edcf8baf2c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:41.721475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47c05e3-aec1-413c-92eb-4edcf8baf2c7 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:41.721894 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47c05e3-aec1-413c-92eb-4edcf8baf2c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:41.722326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c47c05e3-aec1-413c-92eb-4edcf8baf2c7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:41.723149 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c47c05e3-aec1-413c-92eb-4edcf8baf2c7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:41.725827 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c47c05e3-aec1-413c-92eb-4edcf8baf2c7 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:41.742185 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:41.773324 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-de86e839-9da2-4db6-adad-abea8f9db7e4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '39e891d5d8a6431fb6f5343c543214c5'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:41.783415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-de86e839-9da2-4db6-adad-abea8f9db7e4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '39e891d5d8a6431fb6f5343c543214c5'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:41.784773 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-de86e839-9da2-4db6-adad-abea8f9db7e4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '39e891d5d8a6431fb6f5343c543214c5'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:41.785587 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-de86e839-9da2-4db6-adad-abea8f9db7e4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '39e891d5d8a6431fb6f5343c543214c5'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:41.800952 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 863/1756] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:45:41 2022] POST /identity/v3/projects => generated 365 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:45:41.804951 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c47c05e3-aec1-413c-92eb-4edcf8baf2c7 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:41.806951 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-c47c05e3-aec1-413c-92eb-4edcf8baf2c7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:41.826095 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-de86e839-9da2-4db6-adad-abea8f9db7e4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '39e891d5d8a6431fb6f5343c543214c5'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:41.881025 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 894/1757] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:45:41 2022] DELETE /identity/v3/users/39e891d5d8a6431fb6f5343c543214c5 => generated 0 bytes in 443 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:45:41.884575 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf6b7fb-2ebf-4b78-821a-3a72d1838c47 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:41.885011 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf6b7fb-2ebf-4b78-821a-3a72d1838c47 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:41.885395 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf6b7fb-2ebf-4b78-821a-3a72d1838c47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:41.915790 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c47c05e3-aec1-413c-92eb-4edcf8baf2c7 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:41.917885 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-cbf6b7fb-2ebf-4b78-821a-3a72d1838c47 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:41.919286 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-cbf6b7fb-2ebf-4b78-821a-3a72d1838c47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:42.001999 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67956741-835b-414c-9161-3894663773e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:42.002638 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67956741-835b-414c-9161-3894663773e1 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:42.002934 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67956741-835b-414c-9161-3894663773e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:42.003265 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67956741-835b-414c-9161-3894663773e1 admin admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:42.004535 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67956741-835b-414c-9161-3894663773e1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:42.012474 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67956741-835b-414c-9161-3894663773e1 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:42.062446 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 864/1758] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:45:41 2022] POST /identity/v3/users => generated 283 bytes in 256 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:45:42.064601 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67956741-835b-414c-9161-3894663773e1 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:42.083538 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 895/1759] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:45:42.088912 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67956741-835b-414c-9161-3894663773e1 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:42.093935 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54bf1353-ea8c-4ccb-9f25-fbfc3275bfb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:42.094798 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bf1353-ea8c-4ccb-9f25-fbfc3275bfb1 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:42.094987 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bf1353-ea8c-4ccb-9f25-fbfc3275bfb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:42.095458 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bf1353-ea8c-4ccb-9f25-fbfc3275bfb1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:42.096032 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54bf1353-ea8c-4ccb-9f25-fbfc3275bfb1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:42.097806 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54bf1353-ea8c-4ccb-9f25-fbfc3275bfb1 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:42.118130 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 865/1760] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:45:42 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:45:42.123041 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf6b7fb-2ebf-4b78-821a-3a72d1838c47 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:42.142247 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf6b7fb-2ebf-4b78-821a-3a72d1838c47 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:42.147782 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afa60540-efb8-458b-8f1d-c11edf39a66f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:42.148494 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa60540-efb8-458b-8f1d-c11edf39a66f admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:42.148785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa60540-efb8-458b-8f1d-c11edf39a66f admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:42.149066 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa60540-efb8-458b-8f1d-c11edf39a66f admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:42.150355 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afa60540-efb8-458b-8f1d-c11edf39a66f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:42.152090 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afa60540-efb8-458b-8f1d-c11edf39a66f admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:42.167349 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54bf1353-ea8c-4ccb-9f25-fbfc3275bfb1 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:42.169240 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 896/1761] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:45:42 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:45:42.175160 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-54bf1353-ea8c-4ccb-9f25-fbfc3275bfb1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:42.186639 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afa60540-efb8-458b-8f1d-c11edf39a66f admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:42.214127 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afa60540-efb8-458b-8f1d-c11edf39a66f admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:42.221270 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad7ca066-f513-4e57-a63c-772223272e12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:42.222615 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7ca066-f513-4e57-a63c-772223272e12 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:42.225696 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7ca066-f513-4e57-a63c-772223272e12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:42.225696 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7ca066-f513-4e57-a63c-772223272e12 admin admin] PATH_INFO: `/v3/projects/029c537c089d475ab727cc630f290585/users/78c1da510a31474c83c74d560e7776de/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:42.273779 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad7ca066-f513-4e57-a63c-772223272e12 admin admin] RBAC: Authorizing `identity:create_grant(project_id=029c537c089d475ab727cc630f290585, user_id=78c1da510a31474c83c74d560e7776de, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:42.276777 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad7ca066-f513-4e57-a63c-772223272e12 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:42.316469 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 897/1762] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:45:42 2022] PUT /identity/v3/projects/029c537c089d475ab727cc630f290585/users/78c1da510a31474c83c74d560e7776de/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:45:42.317925 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54bf1353-ea8c-4ccb-9f25-fbfc3275bfb1 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:42.320145 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3433a48-4e94-447b-ba78-0a69116a5721 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:42.320507 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3433a48-4e94-447b-ba78-0a69116a5721 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:42.320855 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3433a48-4e94-447b-ba78-0a69116a5721 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:42.355212 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:42.406022 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0108b77b-d703-4ae4-bbbe-22c9d40e2174 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:42.408916 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0108b77b-d703-4ae4-bbbe-22c9d40e2174 service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:42.408916 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0108b77b-d703-4ae4-bbbe-22c9d40e2174 service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:42.409273 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0108b77b-d703-4ae4-bbbe-22c9d40e2174 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:42.422763 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-a3433a48-4e94-447b-ba78-0a69116a5721 None None] MFA Rules not processed for user `78c1da510a31474c83c74d560e7776de`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:42.428240 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a3433a48-4e94-447b-ba78-0a69116a5721 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:42.559772 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0108b77b-d703-4ae4-bbbe-22c9d40e2174 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:42.564551 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:DELETE=8,keystone:SELECT=90,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:42.565547 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0108b77b-d703-4ae4-bbbe-22c9d40e2174 service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:42.597063 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 866/1763] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:45:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 430 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:45:42.621179 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 898/1764] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:42 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 302 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:45:42.627248 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0108b77b-d703-4ae4-bbbe-22c9d40e2174 service neutron] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:42.649774 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0108b77b-d703-4ae4-bbbe-22c9d40e2174 service neutron] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:42.661395 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ce8de1-836f-4d1c-9e51-efaebc360899 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:42.664896 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ce8de1-836f-4d1c-9e51-efaebc360899 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:42.665593 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ce8de1-836f-4d1c-9e51-efaebc360899 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:42.667840 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ce8de1-836f-4d1c-9e51-efaebc360899 admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:42.668559 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ce8de1-836f-4d1c-9e51-efaebc360899 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:42.673051 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ce8de1-836f-4d1c-9e51-efaebc360899 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:42.737171 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 867/1765] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:45:42 2022] POST /identity/v3/projects => generated 365 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:45:42.741588 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3433a48-4e94-447b-ba78-0a69116a5721 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:42.755892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3433a48-4e94-447b-ba78-0a69116a5721 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:42.840601 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf135f6-9fef-4c3c-a040-5bc990e5b220 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:42.841250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf135f6-9fef-4c3c-a040-5bc990e5b220 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:42.841621 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf135f6-9fef-4c3c-a040-5bc990e5b220 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:42.843190 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf135f6-9fef-4c3c-a040-5bc990e5b220 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:42.843190 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cf135f6-9fef-4c3c-a040-5bc990e5b220 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:42.845019 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cf135f6-9fef-4c3c-a040-5bc990e5b220 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:42.879951 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 899/1766] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:45:42 2022] POST /identity/v3/users => generated 289 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:45:42.885970 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ce8de1-836f-4d1c-9e51-efaebc360899 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:42.917586 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ce8de1-836f-4d1c-9e51-efaebc360899 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:42.945001 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea94d585-01bf-46d3-ad55-7a69305b05a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:42.945813 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea94d585-01bf-46d3-ad55-7a69305b05a9 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:42.946190 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea94d585-01bf-46d3-ad55-7a69305b05a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:42.946593 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea94d585-01bf-46d3-ad55-7a69305b05a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:42.947350 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea94d585-01bf-46d3-ad55-7a69305b05a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:42.949196 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea94d585-01bf-46d3-ad55-7a69305b05a9 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:42.993029 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 868/1767] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:45:42 2022] GET /identity/v3/roles => generated 1420 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:45:43.000002 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf135f6-9fef-4c3c-a040-5bc990e5b220 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:43.020627 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf135f6-9fef-4c3c-a040-5bc990e5b220 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:43.031625 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce0c6c1-a8c8-48c1-8c50-872ffdbb3d00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:43.032318 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce0c6c1-a8c8-48c1-8c50-872ffdbb3d00 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:43.032601 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce0c6c1-a8c8-48c1-8c50-872ffdbb3d00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:43.033001 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce0c6c1-a8c8-48c1-8c50-872ffdbb3d00 admin admin] PATH_INFO: `/v3/projects/bbe1c80a60fc48eba50fcf25175fde4f/users/e57f3062752446c8a40a1220e7fa41b7/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:43.064910 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce0c6c1-a8c8-48c1-8c50-872ffdbb3d00 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bbe1c80a60fc48eba50fcf25175fde4f, user_id=e57f3062752446c8a40a1220e7fa41b7, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:43.068993 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fce0c6c1-a8c8-48c1-8c50-872ffdbb3d00 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:43.089453 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 900/1768] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:45:42 2022] PUT /identity/v3/projects/bbe1c80a60fc48eba50fcf25175fde4f/users/e57f3062752446c8a40a1220e7fa41b7/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 89 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:45:43.095819 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea94d585-01bf-46d3-ad55-7a69305b05a9 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:43.108760 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea94d585-01bf-46d3-ad55-7a69305b05a9 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:43.194464 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce0c6c1-a8c8-48c1-8c50-872ffdbb3d00 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:43.196790 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-fce0c6c1-a8c8-48c1-8c50-872ffdbb3d00 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:43.231070 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6fd8d97-873a-41c5-9cd9-ac696b0e4983 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:43.231749 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fd8d97-873a-41c5-9cd9-ac696b0e4983 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:43.232038 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fd8d97-873a-41c5-9cd9-ac696b0e4983 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:43.232451 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fd8d97-873a-41c5-9cd9-ac696b0e4983 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:43.233295 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6fd8d97-873a-41c5-9cd9-ac696b0e4983 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:43.241064 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6fd8d97-873a-41c5-9cd9-ac696b0e4983 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:43.253422 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 869/1769] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:45:43 2022] GET /identity/v3/roles => generated 1420 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:45:43.254929 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6fd8d97-873a-41c5-9cd9-ac696b0e4983 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:43.265584 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6fd8d97-873a-41c5-9cd9-ac696b0e4983 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:43.271914 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a48fc2e-50c2-49a3-978b-fd678e32d900 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:43.272830 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a48fc2e-50c2-49a3-978b-fd678e32d900 admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:43.273278 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a48fc2e-50c2-49a3-978b-fd678e32d900 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:43.273691 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a48fc2e-50c2-49a3-978b-fd678e32d900 admin admin] PATH_INFO: `/v3/projects/bbe1c80a60fc48eba50fcf25175fde4f/users/e57f3062752446c8a40a1220e7fa41b7/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:43.277198 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a48fc2e-50c2-49a3-978b-fd678e32d900 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bbe1c80a60fc48eba50fcf25175fde4f, user_id=e57f3062752446c8a40a1220e7fa41b7, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:43.280659 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a48fc2e-50c2-49a3-978b-fd678e32d900 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:43.293373 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 870/1770] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:45:43 2022] PUT /identity/v3/projects/bbe1c80a60fc48eba50fcf25175fde4f/users/e57f3062752446c8a40a1220e7fa41b7/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:45:43.294347 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fce0c6c1-a8c8-48c1-8c50-872ffdbb3d00 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:43.304369 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0447d9-0408-4ecf-89ce-8d2406063a2c None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:43.304658 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0447d9-0408-4ecf-89ce-8d2406063a2c None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:43.304918 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0447d9-0408-4ecf-89ce-8d2406063a2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:43.347924 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-9b0447d9-0408-4ecf-89ce-8d2406063a2c None None] MFA Rules not processed for user `e57f3062752446c8a40a1220e7fa41b7`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:43.350091 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-9b0447d9-0408-4ecf-89ce-8d2406063a2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:43.372533 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:43.377494 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400919e8-474a-42c4-9986-71aa2ca4adc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:43.378166 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400919e8-474a-42c4-9986-71aa2ca4adc8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:43.378520 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400919e8-474a-42c4-9986-71aa2ca4adc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:43.378805 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400919e8-474a-42c4-9986-71aa2ca4adc8 admin admin] PATH_INFO: `/v3/projects/aa92d710b3a84607a6d958773640b398` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:43.379976 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-400919e8-474a-42c4-9986-71aa2ca4adc8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=aa92d710b3a84607a6d958773640b398)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:43.382499 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-400919e8-474a-42c4-9986-71aa2ca4adc8 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:43.390853 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:43.436966 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 871/1771] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:43 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:45:43.454362 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0447d9-0408-4ecf-89ce-8d2406063a2c None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:43.459409 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-400919e8-474a-42c4-9986-71aa2ca4adc8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'aa92d710b3a84607a6d958773640b398', 'request_id': 'req-400919e8-474a-42c4-9986-71aa2ca4adc8'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:43.461614 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-9b0447d9-0408-4ecf-89ce-8d2406063a2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:43.466716 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-400919e8-474a-42c4-9986-71aa2ca4adc8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'aa92d710b3a84607a6d958773640b398', 'request_id': 'req-400919e8-474a-42c4-9986-71aa2ca4adc8'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:43.482222 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-400919e8-474a-42c4-9986-71aa2ca4adc8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'aa92d710b3a84607a6d958773640b398', 'request_id': 'req-400919e8-474a-42c4-9986-71aa2ca4adc8'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:43.498003 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-400919e8-474a-42c4-9986-71aa2ca4adc8 admin admin] The token cache is being invalidate because project aa92d710b3a84607a6d958773640b398 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:45:43.499003 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-400919e8-474a-42c4-9986-71aa2ca4adc8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:43.499833 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-400919e8-474a-42c4-9986-71aa2ca4adc8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:43.505170 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 901/1772] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:45:43 2022] DELETE /identity/v3/projects/aa92d710b3a84607a6d958773640b398 => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:45:43.508082 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400919e8-474a-42c4-9986-71aa2ca4adc8 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:43.510773 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-400919e8-474a-42c4-9986-71aa2ca4adc8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:43.550660 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0447d9-0408-4ecf-89ce-8d2406063a2c None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:43.612775 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400919e8-474a-42c4-9986-71aa2ca4adc8 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:43.617181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872feba1-f162-4111-81d8-fd19460d846a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:43.617939 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872feba1-f162-4111-81d8-fd19460d846a service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:43.618309 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2093d3-a4ad-4170-8dd1-919055d0923b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:43.618452 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872feba1-f162-4111-81d8-fd19460d846a service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:43.618798 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872feba1-f162-4111-81d8-fd19460d846a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:43.619755 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2093d3-a4ad-4170-8dd1-919055d0923b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:43.619755 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2093d3-a4ad-4170-8dd1-919055d0923b admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:43.619755 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2093d3-a4ad-4170-8dd1-919055d0923b admin admin] PATH_INFO: `/v3/users/fbb4fbdcd1994a98b9a4966fb6adac75` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:43.621223 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-872feba1-f162-4111-81d8-fd19460d846a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:43.624419 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a2093d3-a4ad-4170-8dd1-919055d0923b admin admin] RBAC: Authorizing `identity:delete_user(user_id=fbb4fbdcd1994a98b9a4966fb6adac75)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:43.625463 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a2093d3-a4ad-4170-8dd1-919055d0923b admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:43.680150 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-4a2093d3-a4ad-4170-8dd1-919055d0923b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fbb4fbdcd1994a98b9a4966fb6adac75'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:43.689118 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-4a2093d3-a4ad-4170-8dd1-919055d0923b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fbb4fbdcd1994a98b9a4966fb6adac75'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:43.689812 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-4a2093d3-a4ad-4170-8dd1-919055d0923b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fbb4fbdcd1994a98b9a4966fb6adac75'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:43.690413 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-4a2093d3-a4ad-4170-8dd1-919055d0923b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fbb4fbdcd1994a98b9a4966fb6adac75'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:43.695906 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-872feba1-f162-4111-81d8-fd19460d846a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:43.699249 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-872feba1-f162-4111-81d8-fd19460d846a service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:43.699671 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-4a2093d3-a4ad-4170-8dd1-919055d0923b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fbb4fbdcd1994a98b9a4966fb6adac75'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:43.718602 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 872/1773] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:45:43 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 267 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:45:43.726240 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 902/1774] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:45:43 2022] DELETE /identity/v3/users/fbb4fbdcd1994a98b9a4966fb6adac75 => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:45:43.818329 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872feba1-f162-4111-81d8-fd19460d846a service cinder] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:43.819975 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-872feba1-f162-4111-81d8-fd19460d846a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:43.835754 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872feba1-f162-4111-81d8-fd19460d846a service cinder] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:43.852634 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:43.895541 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757ee9f2-7040-4519-895b-8cdbda8b2d3e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:43.896357 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757ee9f2-7040-4519-895b-8cdbda8b2d3e service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:43.896642 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757ee9f2-7040-4519-895b-8cdbda8b2d3e service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:43.897522 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757ee9f2-7040-4519-895b-8cdbda8b2d3e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:43.900034 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-757ee9f2-7040-4519-895b-8cdbda8b2d3e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:44.001190 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757ee9f2-7040-4519-895b-8cdbda8b2d3e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:44.003407 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757ee9f2-7040-4519-895b-8cdbda8b2d3e service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:44.017904 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 873/1775] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:45:43 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 200 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:45:44.094424 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2093d3-a4ad-4170-8dd1-919055d0923b admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:44.096226 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4a2093d3-a4ad-4170-8dd1-919055d0923b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:44.119281 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:44.209960 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a2093d3-a4ad-4170-8dd1-919055d0923b admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:44.222844 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c7081b9-c183-44b6-bd0f-b5771f50a8d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:44.223508 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7081b9-c183-44b6-bd0f-b5771f50a8d1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:44.224035 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7081b9-c183-44b6-bd0f-b5771f50a8d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:44.224842 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c7081b9-c183-44b6-bd0f-b5771f50a8d1 admin admin] PATH_INFO: `/v3/projects/7f025a2677be439197ad88613957a816` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:44.229707 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c7081b9-c183-44b6-bd0f-b5771f50a8d1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7f025a2677be439197ad88613957a816)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:44.233360 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c7081b9-c183-44b6-bd0f-b5771f50a8d1 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:44.356169 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-6c7081b9-c183-44b6-bd0f-b5771f50a8d1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7f025a2677be439197ad88613957a816', 'request_id': 'req-6c7081b9-c183-44b6-bd0f-b5771f50a8d1'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:44.376213 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-6c7081b9-c183-44b6-bd0f-b5771f50a8d1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7f025a2677be439197ad88613957a816', 'request_id': 'req-6c7081b9-c183-44b6-bd0f-b5771f50a8d1'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:44.416754 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-6c7081b9-c183-44b6-bd0f-b5771f50a8d1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7f025a2677be439197ad88613957a816', 'request_id': 'req-6c7081b9-c183-44b6-bd0f-b5771f50a8d1'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:44.443094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-6c7081b9-c183-44b6-bd0f-b5771f50a8d1 admin admin] The token cache is being invalidate because project 7f025a2677be439197ad88613957a816 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:45:44.443300 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-6c7081b9-c183-44b6-bd0f-b5771f50a8d1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:44.444065 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-6c7081b9-c183-44b6-bd0f-b5771f50a8d1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:44.446208 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 903/1776] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:45:44 2022] DELETE /identity/v3/projects/7f025a2677be439197ad88613957a816 => generated 0 bytes in 352 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:45:44.452806 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d456ba5c-6967-41c6-849d-c1efc525a470 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:44.453900 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d456ba5c-6967-41c6-849d-c1efc525a470 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:44.453900 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d456ba5c-6967-41c6-849d-c1efc525a470 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:44.527376 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-d456ba5c-6967-41c6-849d-c1efc525a470 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:44.529545 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-d456ba5c-6967-41c6-849d-c1efc525a470 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:44.659839 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e3974b-e7af-4428-b06b-5b638e5d619e None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:44.660311 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e3974b-e7af-4428-b06b-5b638e5d619e None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:44.660409 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e3974b-e7af-4428-b06b-5b638e5d619e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:44.675646 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 874/1777] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 225 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:45:44.708112 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-21e3974b-e7af-4428-b06b-5b638e5d619e None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:44.709675 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-21e3974b-e7af-4428-b06b-5b638e5d619e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:44.721489 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 904/1778] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 63 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:45:44.727736 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d456ba5c-6967-41c6-849d-c1efc525a470 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:44.758040 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d456ba5c-6967-41c6-849d-c1efc525a470 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:44.764684 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5571dc7e-e27e-46c3-9d68-03030f6340e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:44.765564 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5571dc7e-e27e-46c3-9d68-03030f6340e0 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:44.765893 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5571dc7e-e27e-46c3-9d68-03030f6340e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:44.767461 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5571dc7e-e27e-46c3-9d68-03030f6340e0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:44.769179 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5571dc7e-e27e-46c3-9d68-03030f6340e0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:45:44.769646 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5571dc7e-e27e-46c3-9d68-03030f6340e0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:44.771933 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5571dc7e-e27e-46c3-9d68-03030f6340e0 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:44.793997 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 875/1779] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:45:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:45:44.795350 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21e3974b-e7af-4428-b06b-5b638e5d619e None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:44.806302 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21e3974b-e7af-4428-b06b-5b638e5d619e None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:44.811699 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71cdc36e-c8b6-429d-862a-e88625af1833 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:44.812794 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cdc36e-c8b6-429d-862a-e88625af1833 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:44.813283 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cdc36e-c8b6-429d-862a-e88625af1833 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:44.813790 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71cdc36e-c8b6-429d-862a-e88625af1833 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:44.814748 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71cdc36e-c8b6-429d-862a-e88625af1833 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:44.817227 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71cdc36e-c8b6-429d-862a-e88625af1833 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:44.857366 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 905/1780] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:45:44 2022] POST /identity/v3/projects => generated 381 bytes in 62 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:45:44.861786 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5571dc7e-e27e-46c3-9d68-03030f6340e0 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:44.876587 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f565459-1473-4246-bd37-c72fd0a0915e None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:44.877014 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f565459-1473-4246-bd37-c72fd0a0915e None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:44.878072 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f565459-1473-4246-bd37-c72fd0a0915e None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:44.880248 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5571dc7e-e27e-46c3-9d68-03030f6340e0 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:44.882196 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 906/1781] 10.222.0.25 () {54 vars in 921 bytes} [Fri Oct 28 09:45:44 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 09:45:44.903944 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801f264b-adc3-4c70-b0bd-462af0608635 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:44.904344 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801f264b-adc3-4c70-b0bd-462af0608635 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:44.904706 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801f264b-adc3-4c70-b0bd-462af0608635 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:44.948517 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-801f264b-adc3-4c70-b0bd-462af0608635 None None] MFA Rules not processed for user `688ceb201d304c2da5373c3cf36c2d79`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:44.950574 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-801f264b-adc3-4c70-b0bd-462af0608635 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:44.995966 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7e6a2e-c81f-4a0b-a5f1-9c8a6f5d5e78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:44.996710 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7e6a2e-c81f-4a0b-a5f1-9c8a6f5d5e78 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:44.996995 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7e6a2e-c81f-4a0b-a5f1-9c8a6f5d5e78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:44.997420 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7e6a2e-c81f-4a0b-a5f1-9c8a6f5d5e78 admin admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:44.998136 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d7e6a2e-c81f-4a0b-a5f1-9c8a6f5d5e78 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:45.000996 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d7e6a2e-c81f-4a0b-a5f1-9c8a6f5d5e78 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:45.010803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:45.031352 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 876/1782] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:45:44 2022] POST /identity/v3/users => generated 291 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:45:45.035824 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7e6a2e-c81f-4a0b-a5f1-9c8a6f5d5e78 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:45.051172 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 907/1783] 10.222.0.25 () {58 vars in 1018 bytes} [Fri Oct 28 09:45:44 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:45:45.052197 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7e6a2e-c81f-4a0b-a5f1-9c8a6f5d5e78 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:45.059500 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387405d1-0ee5-4ee1-95a0-7e96d52faef8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:45.060378 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387405d1-0ee5-4ee1-95a0-7e96d52faef8 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:45.060879 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387405d1-0ee5-4ee1-95a0-7e96d52faef8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:45.061426 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-387405d1-0ee5-4ee1-95a0-7e96d52faef8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:45.062286 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-387405d1-0ee5-4ee1-95a0-7e96d52faef8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:45.064213 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-387405d1-0ee5-4ee1-95a0-7e96d52faef8 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:45.069501 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801f264b-adc3-4c70-b0bd-462af0608635 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:45.071015 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-801f264b-adc3-4c70-b0bd-462af0608635 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:45.077131 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 877/1784] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:45:45 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:45:45.083435 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387405d1-0ee5-4ee1-95a0-7e96d52faef8 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:45.093316 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801f264b-adc3-4c70-b0bd-462af0608635 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:45.095400 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-387405d1-0ee5-4ee1-95a0-7e96d52faef8 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:45.099556 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44815349-70f0-4894-8aac-c69d0fbdca5b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:45.100189 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44815349-70f0-4894-8aac-c69d0fbdca5b service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:45.100478 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44815349-70f0-4894-8aac-c69d0fbdca5b service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:45.100760 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44815349-70f0-4894-8aac-c69d0fbdca5b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:45.100969 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea0a7d64-41d1-467d-86af-e4a274c1d538 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:45.101635 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0a7d64-41d1-467d-86af-e4a274c1d538 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:45.101871 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0a7d64-41d1-467d-86af-e4a274c1d538 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:45.102149 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0a7d64-41d1-467d-86af-e4a274c1d538 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:45.102835 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea0a7d64-41d1-467d-86af-e4a274c1d538 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:45.104824 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea0a7d64-41d1-467d-86af-e4a274c1d538 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:45.107497 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:45.112179 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44815349-70f0-4894-8aac-c69d0fbdca5b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:45.114419 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44815349-70f0-4894-8aac-c69d0fbdca5b service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:45.119466 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 878/1785] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:45:45 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:45:45.121338 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea0a7d64-41d1-467d-86af-e4a274c1d538 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:45.130176 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 908/1786] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:45:45 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 62 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:45:45.137031 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea0a7d64-41d1-467d-86af-e4a274c1d538 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:45.141989 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8d034b-5181-4820-a8fb-0f74dec77f3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:45.142894 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8d034b-5181-4820-a8fb-0f74dec77f3a admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:45.143867 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8d034b-5181-4820-a8fb-0f74dec77f3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:45.143867 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8d034b-5181-4820-a8fb-0f74dec77f3a admin admin] PATH_INFO: `/v3/projects/70c97375d3044a5090d53fa6d25bb957/users/6509a502ce1745c79a5bd28274760899/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:45.164934 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8d034b-5181-4820-a8fb-0f74dec77f3a admin admin] RBAC: Authorizing `identity:create_grant(project_id=70c97375d3044a5090d53fa6d25bb957, user_id=6509a502ce1745c79a5bd28274760899, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:45.168888 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8d034b-5181-4820-a8fb-0f74dec77f3a admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:45.191968 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 879/1787] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:45:45 2022] PUT /identity/v3/projects/70c97375d3044a5090d53fa6d25bb957/users/6509a502ce1745c79a5bd28274760899/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:45:45.204917 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20cc31dc-5c6c-472e-a65f-82c88d6fd955 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:45.205220 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20cc31dc-5c6c-472e-a65f-82c88d6fd955 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:45.205511 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20cc31dc-5c6c-472e-a65f-82c88d6fd955 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:45.245154 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-20cc31dc-5c6c-472e-a65f-82c88d6fd955 None None] MFA Rules not processed for user `6509a502ce1745c79a5bd28274760899`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:45.246560 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-20cc31dc-5c6c-472e-a65f-82c88d6fd955 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:45.359250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 909/1788] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:45 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:45:45.363933 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8d034b-5181-4820-a8fb-0f74dec77f3a admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:45.465360 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8d034b-5181-4820-a8fb-0f74dec77f3a admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:45.474111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1328c3-b624-4ef2-8c38-c16375b38b7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:45.475431 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1328c3-b624-4ef2-8c38-c16375b38b7e admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:45.475868 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1328c3-b624-4ef2-8c38-c16375b38b7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:45.476470 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1328c3-b624-4ef2-8c38-c16375b38b7e admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:45.477852 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f1328c3-b624-4ef2-8c38-c16375b38b7e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:45.480648 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f1328c3-b624-4ef2-8c38-c16375b38b7e admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:45.517505 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 880/1789] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:45:45 2022] POST /identity/v3/projects => generated 381 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:45:45.522186 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20cc31dc-5c6c-472e-a65f-82c88d6fd955 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:45.534211 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20cc31dc-5c6c-472e-a65f-82c88d6fd955 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:45.628875 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727d896d-839f-4ffb-b6b4-46dc99a9adba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:45.629839 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727d896d-839f-4ffb-b6b4-46dc99a9adba admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:45.629839 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727d896d-839f-4ffb-b6b4-46dc99a9adba admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:45.630097 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727d896d-839f-4ffb-b6b4-46dc99a9adba admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:45.631603 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-727d896d-839f-4ffb-b6b4-46dc99a9adba admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:45.633527 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-727d896d-839f-4ffb-b6b4-46dc99a9adba admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:45.665836 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 910/1790] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:45:45 2022] POST /identity/v3/users => generated 297 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:45:45.669746 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1328c3-b624-4ef2-8c38-c16375b38b7e admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:45.682977 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1328c3-b624-4ef2-8c38-c16375b38b7e admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:45.687846 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-026b5fe0-3f07-4f2a-ae51-b43cbbcabc0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:45.688745 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026b5fe0-3f07-4f2a-ae51-b43cbbcabc0f admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:45.689238 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026b5fe0-3f07-4f2a-ae51-b43cbbcabc0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:45.689810 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026b5fe0-3f07-4f2a-ae51-b43cbbcabc0f admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:45.690950 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-026b5fe0-3f07-4f2a-ae51-b43cbbcabc0f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:45.693724 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-026b5fe0-3f07-4f2a-ae51-b43cbbcabc0f admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:45.738561 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 881/1791] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:45:45 2022] GET /identity/v3/roles => generated 1420 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:45:45.739110 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727d896d-839f-4ffb-b6b4-46dc99a9adba admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:45.756250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727d896d-839f-4ffb-b6b4-46dc99a9adba admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:45.763077 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d63a43e-200c-4ff6-99e9-4c636f3db725 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:45.764042 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d63a43e-200c-4ff6-99e9-4c636f3db725 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:45.764519 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d63a43e-200c-4ff6-99e9-4c636f3db725 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:45.764917 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d63a43e-200c-4ff6-99e9-4c636f3db725 admin admin] PATH_INFO: `/v3/projects/c495011bcbf64285a8961a5920b89934/users/1c5a105ebf334a5b9cbf09d809db78aa/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:45.786217 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d63a43e-200c-4ff6-99e9-4c636f3db725 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c495011bcbf64285a8961a5920b89934, user_id=1c5a105ebf334a5b9cbf09d809db78aa, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:45.789013 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d63a43e-200c-4ff6-99e9-4c636f3db725 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:45.816347 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 911/1792] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:45:45 2022] PUT /identity/v3/projects/c495011bcbf64285a8961a5920b89934/users/1c5a105ebf334a5b9cbf09d809db78aa/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:45:45.821432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-026b5fe0-3f07-4f2a-ae51-b43cbbcabc0f admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:45.832937 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-026b5fe0-3f07-4f2a-ae51-b43cbbcabc0f admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:45.900697 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729eb756-d42c-48d6-a9c5-3d8a614c095f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:45.901323 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729eb756-d42c-48d6-a9c5-3d8a614c095f admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:45.901643 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729eb756-d42c-48d6-a9c5-3d8a614c095f admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:45.902045 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729eb756-d42c-48d6-a9c5-3d8a614c095f admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:45.902987 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-729eb756-d42c-48d6-a9c5-3d8a614c095f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:45.905626 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-729eb756-d42c-48d6-a9c5-3d8a614c095f admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:45.930654 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 882/1793] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:45:45 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:45:45.930868 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d63a43e-200c-4ff6-99e9-4c636f3db725 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:45.946999 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d63a43e-200c-4ff6-99e9-4c636f3db725 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:45.968622 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57067817-39be-4004-b840-6fcffbb0319c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:45.969309 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57067817-39be-4004-b840-6fcffbb0319c admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:45.969588 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57067817-39be-4004-b840-6fcffbb0319c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:45.969861 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57067817-39be-4004-b840-6fcffbb0319c admin admin] PATH_INFO: `/v3/projects/c495011bcbf64285a8961a5920b89934/users/1c5a105ebf334a5b9cbf09d809db78aa/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:45.975798 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57067817-39be-4004-b840-6fcffbb0319c admin admin] RBAC: Authorizing `identity:create_grant(project_id=c495011bcbf64285a8961a5920b89934, user_id=1c5a105ebf334a5b9cbf09d809db78aa, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:45.978459 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57067817-39be-4004-b840-6fcffbb0319c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:45.991809 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 912/1794] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:45:45 2022] PUT /identity/v3/projects/c495011bcbf64285a8961a5920b89934/users/1c5a105ebf334a5b9cbf09d809db78aa/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:45:46.002201 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787116fc-007a-44bb-95df-2ad91b8feb2b None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:46.002520 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787116fc-007a-44bb-95df-2ad91b8feb2b None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:46.002938 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-787116fc-007a-44bb-95df-2ad91b8feb2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:46.031483 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:INSERT=5,keystone:SELECT=95 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:46.034804 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-787116fc-007a-44bb-95df-2ad91b8feb2b None None] MFA Rules not processed for user `1c5a105ebf334a5b9cbf09d809db78aa`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:46.037365 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-787116fc-007a-44bb-95df-2ad91b8feb2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:46.124626 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 883/1795] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:46 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:45:46.135781 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57067817-39be-4004-b840-6fcffbb0319c admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:46.137489 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-57067817-39be-4004-b840-6fcffbb0319c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:46.201116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:46.226424 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57067817-39be-4004-b840-6fcffbb0319c admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:46.232955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8585da08-194c-4b01-a314-0cbf9c348a4c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:46.233813 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8585da08-194c-4b01-a314-0cbf9c348a4c service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:46.234604 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8585da08-194c-4b01-a314-0cbf9c348a4c service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:46.235100 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8585da08-194c-4b01-a314-0cbf9c348a4c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:46.247810 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8585da08-194c-4b01-a314-0cbf9c348a4c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:46.249923 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8585da08-194c-4b01-a314-0cbf9c348a4c service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:46.263470 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 913/1796] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:45:46 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:45:46.375939 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-787116fc-007a-44bb-95df-2ad91b8feb2b None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:46.388816 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-787116fc-007a-44bb-95df-2ad91b8feb2b None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:46.392548 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c205295-35ff-486a-98c5-05ae0efac1c9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:46.393174 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c205295-35ff-486a-98c5-05ae0efac1c9 service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:46.393453 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c205295-35ff-486a-98c5-05ae0efac1c9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:46.393869 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c205295-35ff-486a-98c5-05ae0efac1c9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:46.473865 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c205295-35ff-486a-98c5-05ae0efac1c9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:46.475392 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c205295-35ff-486a-98c5-05ae0efac1c9 service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:46.485227 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 884/1797] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:45:46 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:45:48.229473 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8585da08-194c-4b01-a314-0cbf9c348a4c service cinder] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:48.231151 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-8585da08-194c-4b01-a314-0cbf9c348a4c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:48.330968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8585da08-194c-4b01-a314-0cbf9c348a4c service cinder] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:48.360585 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f801d4-39c5-4949-96ca-f89b8b6c967c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:48.361617 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f801d4-39c5-4949-96ca-f89b8b6c967c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:48.362047 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f801d4-39c5-4949-96ca-f89b8b6c967c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:48.362449 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f801d4-39c5-4949-96ca-f89b8b6c967c admin admin] PATH_INFO: `/v3/users/78c1da510a31474c83c74d560e7776de` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:48.364608 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f801d4-39c5-4949-96ca-f89b8b6c967c admin admin] RBAC: Authorizing `identity:delete_user(user_id=78c1da510a31474c83c74d560e7776de)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:48.367237 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f801d4-39c5-4949-96ca-f89b8b6c967c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:48.437980 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-56f801d4-39c5-4949-96ca-f89b8b6c967c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '78c1da510a31474c83c74d560e7776de'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:48.448143 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-56f801d4-39c5-4949-96ca-f89b8b6c967c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '78c1da510a31474c83c74d560e7776de'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:48.449057 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-56f801d4-39c5-4949-96ca-f89b8b6c967c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '78c1da510a31474c83c74d560e7776de'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:48.449746 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-56f801d4-39c5-4949-96ca-f89b8b6c967c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '78c1da510a31474c83c74d560e7776de'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:48.468046 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-56f801d4-39c5-4949-96ca-f89b8b6c967c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '78c1da510a31474c83c74d560e7776de'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:48.486778 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 914/1798] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:45:48 2022] DELETE /identity/v3/users/78c1da510a31474c83c74d560e7776de => generated 0 bytes in 258 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:45:48.491969 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0c4a59-6838-4dbf-b5ed-7e53feb3fcbb None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:48.492456 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0c4a59-6838-4dbf-b5ed-7e53feb3fcbb None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:48.492545 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0c4a59-6838-4dbf-b5ed-7e53feb3fcbb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:48.512746 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-6c0c4a59-6838-4dbf-b5ed-7e53feb3fcbb None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:48.514025 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-6c0c4a59-6838-4dbf-b5ed-7e53feb3fcbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:48.580027 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 885/1799] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:45:48.589913 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f801d4-39c5-4949-96ca-f89b8b6c967c admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:48.591837 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-56f801d4-39c5-4949-96ca-f89b8b6c967c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:48.655309 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f801d4-39c5-4949-96ca-f89b8b6c967c admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:48.659511 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30f69a1-da1d-4f3f-ae3e-05c880d097f3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:48.660121 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30f69a1-da1d-4f3f-ae3e-05c880d097f3 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:48.660407 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30f69a1-da1d-4f3f-ae3e-05c880d097f3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:48.660702 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b30f69a1-da1d-4f3f-ae3e-05c880d097f3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:48.679367 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b30f69a1-da1d-4f3f-ae3e-05c880d097f3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:48.682339 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b30f69a1-da1d-4f3f-ae3e-05c880d097f3 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:48.689434 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:48.696802 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 915/1800] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:45:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:45:49.167362 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c0c4a59-6838-4dbf-b5ed-7e53feb3fcbb None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:49.168918 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-6c0c4a59-6838-4dbf-b5ed-7e53feb3fcbb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:49.193582 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c0c4a59-6838-4dbf-b5ed-7e53feb3fcbb None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:49.200562 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8133c23-13ba-4406-bb4a-ec5d8515ead9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:49.201566 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8133c23-13ba-4406-bb4a-ec5d8515ead9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:49.201950 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8133c23-13ba-4406-bb4a-ec5d8515ead9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:49.202425 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8133c23-13ba-4406-bb4a-ec5d8515ead9 admin admin] PATH_INFO: `/v3/projects/029c537c089d475ab727cc630f290585` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:49.203925 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8133c23-13ba-4406-bb4a-ec5d8515ead9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=029c537c089d475ab727cc630f290585)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:49.205852 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8133c23-13ba-4406-bb4a-ec5d8515ead9 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:49.242526 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:49.256092 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-c8133c23-13ba-4406-bb4a-ec5d8515ead9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '029c537c089d475ab727cc630f290585', 'request_id': 'req-c8133c23-13ba-4406-bb4a-ec5d8515ead9'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:49.262221 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-c8133c23-13ba-4406-bb4a-ec5d8515ead9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '029c537c089d475ab727cc630f290585', 'request_id': 'req-c8133c23-13ba-4406-bb4a-ec5d8515ead9'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:49.271660 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-c8133c23-13ba-4406-bb4a-ec5d8515ead9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '029c537c089d475ab727cc630f290585', 'request_id': 'req-c8133c23-13ba-4406-bb4a-ec5d8515ead9'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:49.282326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-c8133c23-13ba-4406-bb4a-ec5d8515ead9 admin admin] The token cache is being invalidate because project 029c537c089d475ab727cc630f290585 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:45:49.282659 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-c8133c23-13ba-4406-bb4a-ec5d8515ead9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:49.283313 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-c8133c23-13ba-4406-bb4a-ec5d8515ead9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:49.284819 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 886/1801] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:45:49 2022] DELETE /identity/v3/projects/029c537c089d475ab727cc630f290585 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:45:49.289006 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30f69a1-da1d-4f3f-ae3e-05c880d097f3 service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:49.299619 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b30f69a1-da1d-4f3f-ae3e-05c880d097f3 service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:49.357945 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87d2d66b-0e86-4207-8fd8-764de4c4c0f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:49.358617 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d2d66b-0e86-4207-8fd8-764de4c4c0f2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:49.358898 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d2d66b-0e86-4207-8fd8-764de4c4c0f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:49.359171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87d2d66b-0e86-4207-8fd8-764de4c4c0f2 admin admin] PATH_INFO: `/v3/users/e57f3062752446c8a40a1220e7fa41b7` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:49.360293 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d2d66b-0e86-4207-8fd8-764de4c4c0f2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e57f3062752446c8a40a1220e7fa41b7)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:49.362214 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87d2d66b-0e86-4207-8fd8-764de4c4c0f2 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:49.416405 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-87d2d66b-0e86-4207-8fd8-764de4c4c0f2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e57f3062752446c8a40a1220e7fa41b7'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:49.424552 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-87d2d66b-0e86-4207-8fd8-764de4c4c0f2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e57f3062752446c8a40a1220e7fa41b7'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:49.425536 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-87d2d66b-0e86-4207-8fd8-764de4c4c0f2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e57f3062752446c8a40a1220e7fa41b7'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:49.426155 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-87d2d66b-0e86-4207-8fd8-764de4c4c0f2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e57f3062752446c8a40a1220e7fa41b7'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:49.434613 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-87d2d66b-0e86-4207-8fd8-764de4c4c0f2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e57f3062752446c8a40a1220e7fa41b7'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:49.458128 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 916/1802] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:45:49 2022] DELETE /identity/v3/users/e57f3062752446c8a40a1220e7fa41b7 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:45:49.777219 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8133c23-13ba-4406-bb4a-ec5d8515ead9 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:49.779236 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-c8133c23-13ba-4406-bb4a-ec5d8515ead9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:49.792709 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8133c23-13ba-4406-bb4a-ec5d8515ead9 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:49.859987 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6511f5a9-29aa-49e2-8b69-cc755a3f6b15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:49.860634 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6511f5a9-29aa-49e2-8b69-cc755a3f6b15 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:49.860917 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6511f5a9-29aa-49e2-8b69-cc755a3f6b15 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:49.861213 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6511f5a9-29aa-49e2-8b69-cc755a3f6b15 admin admin] PATH_INFO: `/v3/projects/bbe1c80a60fc48eba50fcf25175fde4f` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:49.862319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6511f5a9-29aa-49e2-8b69-cc755a3f6b15 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bbe1c80a60fc48eba50fcf25175fde4f)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:49.864264 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6511f5a9-29aa-49e2-8b69-cc755a3f6b15 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:49.914894 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-6511f5a9-29aa-49e2-8b69-cc755a3f6b15 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bbe1c80a60fc48eba50fcf25175fde4f', 'request_id': 'req-6511f5a9-29aa-49e2-8b69-cc755a3f6b15'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:49.921483 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-6511f5a9-29aa-49e2-8b69-cc755a3f6b15 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bbe1c80a60fc48eba50fcf25175fde4f', 'request_id': 'req-6511f5a9-29aa-49e2-8b69-cc755a3f6b15'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:49.931347 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-6511f5a9-29aa-49e2-8b69-cc755a3f6b15 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bbe1c80a60fc48eba50fcf25175fde4f', 'request_id': 'req-6511f5a9-29aa-49e2-8b69-cc755a3f6b15'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:49.942075 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-6511f5a9-29aa-49e2-8b69-cc755a3f6b15 admin admin] The token cache is being invalidate because project bbe1c80a60fc48eba50fcf25175fde4f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:45:49.942418 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-6511f5a9-29aa-49e2-8b69-cc755a3f6b15 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:49.943115 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-6511f5a9-29aa-49e2-8b69-cc755a3f6b15 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:49.944643 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 887/1803] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:45:49 2022] DELETE /identity/v3/projects/bbe1c80a60fc48eba50fcf25175fde4f => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:45:49.950816 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45464d9-3a07-4977-9b89-5be059da2177 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:49.951133 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45464d9-3a07-4977-9b89-5be059da2177 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:49.951414 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e45464d9-3a07-4977-9b89-5be059da2177 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:49.973520 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-e45464d9-3a07-4977-9b89-5be059da2177 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:49.975304 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e45464d9-3a07-4977-9b89-5be059da2177 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:50.041600 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 917/1804] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:45:50.049085 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730be4b5-625e-4d9f-9fbb-f16c2e81aa07 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:50.049369 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730be4b5-625e-4d9f-9fbb-f16c2e81aa07 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:50.049648 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730be4b5-625e-4d9f-9fbb-f16c2e81aa07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:50.071380 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-730be4b5-625e-4d9f-9fbb-f16c2e81aa07 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:50.073032 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-730be4b5-625e-4d9f-9fbb-f16c2e81aa07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:50.081480 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 888/1805] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:45:50.085599 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45464d9-3a07-4977-9b89-5be059da2177 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:50.096211 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e45464d9-3a07-4977-9b89-5be059da2177 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:50.102904 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-752b5b92-8238-4c9e-8a8c-a90c8963f02d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:50.103243 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752b5b92-8238-4c9e-8a8c-a90c8963f02d admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:50.103521 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752b5b92-8238-4c9e-8a8c-a90c8963f02d admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:50.103798 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752b5b92-8238-4c9e-8a8c-a90c8963f02d admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:50.104451 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-752b5b92-8238-4c9e-8a8c-a90c8963f02d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:45:50.104758 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-752b5b92-8238-4c9e-8a8c-a90c8963f02d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:50.106411 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-752b5b92-8238-4c9e-8a8c-a90c8963f02d admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:50.120692 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 918/1806] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:45:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:45:50.124793 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730be4b5-625e-4d9f-9fbb-f16c2e81aa07 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:50.135666 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730be4b5-625e-4d9f-9fbb-f16c2e81aa07 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:50.150941 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49a7603-a299-4259-b3bc-5d5929113f7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:50.151722 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49a7603-a299-4259-b3bc-5d5929113f7a admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:50.152634 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49a7603-a299-4259-b3bc-5d5929113f7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:50.152634 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49a7603-a299-4259-b3bc-5d5929113f7a admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:50.152937 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f49a7603-a299-4259-b3bc-5d5929113f7a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:50.154894 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f49a7603-a299-4259-b3bc-5d5929113f7a admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:50.185087 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 889/1807] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:45:50 2022] POST /identity/v3/projects => generated 373 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:45:50.188980 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-752b5b92-8238-4c9e-8a8c-a90c8963f02d admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:50.199256 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-752b5b92-8238-4c9e-8a8c-a90c8963f02d admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:50.203290 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:50.276456 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8577a5-5aff-4ec0-885f-902828b49e5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:50.277161 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8577a5-5aff-4ec0-885f-902828b49e5e admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:50.277515 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8577a5-5aff-4ec0-885f-902828b49e5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:50.277866 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8577a5-5aff-4ec0-885f-902828b49e5e admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:50.278610 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a8577a5-5aff-4ec0-885f-902828b49e5e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:50.280767 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a8577a5-5aff-4ec0-885f-902828b49e5e admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:50.304792 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 919/1808] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:45:50 2022] POST /identity/v3/users => generated 287 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:45:50.309059 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49a7603-a299-4259-b3bc-5d5929113f7a admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:50.319698 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49a7603-a299-4259-b3bc-5d5929113f7a admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:50.324468 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c9d3ee-a595-4db9-bc0d-6f57190862bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:50.325084 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c9d3ee-a595-4db9-bc0d-6f57190862bc admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:50.325367 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c9d3ee-a595-4db9-bc0d-6f57190862bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:50.325642 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c9d3ee-a595-4db9-bc0d-6f57190862bc admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:50.326291 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4c9d3ee-a595-4db9-bc0d-6f57190862bc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:50.327924 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4c9d3ee-a595-4db9-bc0d-6f57190862bc admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:50.339037 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 890/1809] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:45:50 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:45:50.343064 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8577a5-5aff-4ec0-885f-902828b49e5e admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:50.353429 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8577a5-5aff-4ec0-885f-902828b49e5e admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:50.358551 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a168e99-1a1d-4398-9c68-ee4c6894a743 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:50.359166 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a168e99-1a1d-4398-9c68-ee4c6894a743 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:50.359448 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a168e99-1a1d-4398-9c68-ee4c6894a743 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:50.359723 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a168e99-1a1d-4398-9c68-ee4c6894a743 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:50.360370 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a168e99-1a1d-4398-9c68-ee4c6894a743 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:50.361966 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a168e99-1a1d-4398-9c68-ee4c6894a743 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:50.373838 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 920/1810] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:45:50 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:45:50.377848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c9d3ee-a595-4db9-bc0d-6f57190862bc admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:50.387775 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c9d3ee-a595-4db9-bc0d-6f57190862bc admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:50.392399 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638337c7-d67d-457c-a510-d22ed47dde38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:50.393070 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638337c7-d67d-457c-a510-d22ed47dde38 admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:50.393336 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638337c7-d67d-457c-a510-d22ed47dde38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:50.393612 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638337c7-d67d-457c-a510-d22ed47dde38 admin admin] PATH_INFO: `/v3/projects/850015556bfb49fe8f1acef5a8eb81da/users/f91fa09755ac4369abcc8a025e6525ae/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:50.402928 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=85,keystone:DELETE=12,keystone:INSERT=3 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:50.410290 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-638337c7-d67d-457c-a510-d22ed47dde38 admin admin] RBAC: Authorizing `identity:create_grant(project_id=850015556bfb49fe8f1acef5a8eb81da, user_id=f91fa09755ac4369abcc8a025e6525ae, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:50.414010 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-638337c7-d67d-457c-a510-d22ed47dde38 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:50.426811 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 891/1811] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:45:50 2022] PUT /identity/v3/projects/850015556bfb49fe8f1acef5a8eb81da/users/f91fa09755ac4369abcc8a025e6525ae/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:45:50.438667 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa3a289-7fef-4de7-9191-e0d45f9d3fef None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:50.438891 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa3a289-7fef-4de7-9191-e0d45f9d3fef None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:50.439164 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa3a289-7fef-4de7-9191-e0d45f9d3fef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:50.466676 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-ffa3a289-7fef-4de7-9191-e0d45f9d3fef None None] MFA Rules not processed for user `f91fa09755ac4369abcc8a025e6525ae`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:50.467921 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ffa3a289-7fef-4de7-9191-e0d45f9d3fef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:50.533106 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 921/1812] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:50 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:45:50.538481 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638337c7-d67d-457c-a510-d22ed47dde38 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:50.609330 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638337c7-d67d-457c-a510-d22ed47dde38 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:50.614188 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f58ba14-3d0c-42b1-94c5-1eba650fd978 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:50.614851 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f58ba14-3d0c-42b1-94c5-1eba650fd978 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:50.615132 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f58ba14-3d0c-42b1-94c5-1eba650fd978 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:50.615456 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f58ba14-3d0c-42b1-94c5-1eba650fd978 admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:50.616055 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f58ba14-3d0c-42b1-94c5-1eba650fd978 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:50.618276 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f58ba14-3d0c-42b1-94c5-1eba650fd978 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:50.645500 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 892/1813] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:45:50 2022] POST /identity/v3/projects => generated 373 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:45:50.650433 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa3a289-7fef-4de7-9191-e0d45f9d3fef None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:50.661136 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa3a289-7fef-4de7-9191-e0d45f9d3fef None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:50.718544 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ccff3d0-b04d-471f-b443-7794e90243bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:50.719081 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ccff3d0-b04d-471f-b443-7794e90243bf admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:50.719362 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ccff3d0-b04d-471f-b443-7794e90243bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:50.719636 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ccff3d0-b04d-471f-b443-7794e90243bf admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:50.720269 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ccff3d0-b04d-471f-b443-7794e90243bf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:50.722180 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ccff3d0-b04d-471f-b443-7794e90243bf admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:50.744805 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 922/1814] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:45:50 2022] POST /identity/v3/users => generated 293 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:45:50.748853 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f58ba14-3d0c-42b1-94c5-1eba650fd978 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:50.760067 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f58ba14-3d0c-42b1-94c5-1eba650fd978 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:50.765384 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c89cc1-851c-4178-840d-d2a9f26ba63a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:50.766004 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c89cc1-851c-4178-840d-d2a9f26ba63a admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:50.766283 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c89cc1-851c-4178-840d-d2a9f26ba63a admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:50.766596 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c89cc1-851c-4178-840d-d2a9f26ba63a admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:50.767246 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48c89cc1-851c-4178-840d-d2a9f26ba63a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:50.768853 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48c89cc1-851c-4178-840d-d2a9f26ba63a admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:50.780936 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 893/1815] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:45:50 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:45:50.785017 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ccff3d0-b04d-471f-b443-7794e90243bf admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:50.791812 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:50.795851 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ccff3d0-b04d-471f-b443-7794e90243bf admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:50.801394 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b60301d-a0b4-4d96-b77a-84bed700a2e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:50.802060 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b60301d-a0b4-4d96-b77a-84bed700a2e2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:50.802439 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b60301d-a0b4-4d96-b77a-84bed700a2e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:50.802729 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b60301d-a0b4-4d96-b77a-84bed700a2e2 admin admin] PATH_INFO: `/v3/projects/b6f3f3448a024da9b80d46c67db75454/users/7b1a5fd5976f46f8ad05193af4ab879b/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:50.817573 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b60301d-a0b4-4d96-b77a-84bed700a2e2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b6f3f3448a024da9b80d46c67db75454, user_id=7b1a5fd5976f46f8ad05193af4ab879b, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:50.820870 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b60301d-a0b4-4d96-b77a-84bed700a2e2 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:50.835236 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 923/1816] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:45:50 2022] PUT /identity/v3/projects/b6f3f3448a024da9b80d46c67db75454/users/7b1a5fd5976f46f8ad05193af4ab879b/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:45:50.839548 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c89cc1-851c-4178-840d-d2a9f26ba63a admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:50.849971 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c89cc1-851c-4178-840d-d2a9f26ba63a admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:50.906784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4c701c-6e2f-49d1-939c-6aae008e2ff0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:50.907394 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4c701c-6e2f-49d1-939c-6aae008e2ff0 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:50.907673 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4c701c-6e2f-49d1-939c-6aae008e2ff0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:50.907948 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4c701c-6e2f-49d1-939c-6aae008e2ff0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:50.908584 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d4c701c-6e2f-49d1-939c-6aae008e2ff0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:50.910164 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d4c701c-6e2f-49d1-939c-6aae008e2ff0 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:50.920987 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 894/1817] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:45:50 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:45:50.925245 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b60301d-a0b4-4d96-b77a-84bed700a2e2 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:50.935941 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b60301d-a0b4-4d96-b77a-84bed700a2e2 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:50.940777 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac2218d-fd41-499e-b533-c3e30a78443c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:50.941428 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac2218d-fd41-499e-b533-c3e30a78443c admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:50.941709 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac2218d-fd41-499e-b533-c3e30a78443c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:50.941984 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac2218d-fd41-499e-b533-c3e30a78443c admin admin] PATH_INFO: `/v3/projects/b6f3f3448a024da9b80d46c67db75454/users/7b1a5fd5976f46f8ad05193af4ab879b/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:50.944525 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac2218d-fd41-499e-b533-c3e30a78443c admin admin] RBAC: Authorizing `identity:create_grant(project_id=b6f3f3448a024da9b80d46c67db75454, user_id=7b1a5fd5976f46f8ad05193af4ab879b, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:50.947229 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac2218d-fd41-499e-b533-c3e30a78443c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:50.957765 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 924/1818] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:45:50 2022] PUT /identity/v3/projects/b6f3f3448a024da9b80d46c67db75454/users/7b1a5fd5976f46f8ad05193af4ab879b/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:45:50.968403 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a799bcb8-8817-476e-a700-ccb67b62d6e6 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:50.968770 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a799bcb8-8817-476e-a700-ccb67b62d6e6 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:50.968989 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a799bcb8-8817-476e-a700-ccb67b62d6e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:51.002373 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-a799bcb8-8817-476e-a700-ccb67b62d6e6 None None] MFA Rules not processed for user `7b1a5fd5976f46f8ad05193af4ab879b`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:51.003655 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-a799bcb8-8817-476e-a700-ccb67b62d6e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:51.059135 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:51.092473 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 895/1819] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:50 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:45:51.104586 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac2218d-fd41-499e-b533-c3e30a78443c admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:51.107749 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-fac2218d-fd41-499e-b533-c3e30a78443c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:51.180450 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac2218d-fd41-499e-b533-c3e30a78443c admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:51.184920 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7258d96e-88f1-46d3-bd4b-f36e2974aa12 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:51.185595 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7258d96e-88f1-46d3-bd4b-f36e2974aa12 service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:51.185863 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7258d96e-88f1-46d3-bd4b-f36e2974aa12 service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:51.186136 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7258d96e-88f1-46d3-bd4b-f36e2974aa12 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:51.259730 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7258d96e-88f1-46d3-bd4b-f36e2974aa12 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:51.261404 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7258d96e-88f1-46d3-bd4b-f36e2974aa12 service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:51.278903 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 925/1820] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:45:51 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 175 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:45:52.372469 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a799bcb8-8817-476e-a700-ccb67b62d6e6 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:52.374135 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-a799bcb8-8817-476e-a700-ccb67b62d6e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:52.463675 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a799bcb8-8817-476e-a700-ccb67b62d6e6 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:52.468240 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462eb941-7d20-406a-8d2c-7a7c29ad84db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:52.468883 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462eb941-7d20-406a-8d2c-7a7c29ad84db admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:52.469162 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462eb941-7d20-406a-8d2c-7a7c29ad84db admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:52.469435 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462eb941-7d20-406a-8d2c-7a7c29ad84db admin admin] PATH_INFO: `/v3/users/6509a502ce1745c79a5bd28274760899` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:52.470617 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462eb941-7d20-406a-8d2c-7a7c29ad84db admin admin] RBAC: Authorizing `identity:delete_user(user_id=6509a502ce1745c79a5bd28274760899)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:52.472530 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462eb941-7d20-406a-8d2c-7a7c29ad84db admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:52.535285 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-462eb941-7d20-406a-8d2c-7a7c29ad84db admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6509a502ce1745c79a5bd28274760899'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:52.543978 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-462eb941-7d20-406a-8d2c-7a7c29ad84db admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6509a502ce1745c79a5bd28274760899'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:52.544862 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-462eb941-7d20-406a-8d2c-7a7c29ad84db admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6509a502ce1745c79a5bd28274760899'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:52.545469 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-462eb941-7d20-406a-8d2c-7a7c29ad84db admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6509a502ce1745c79a5bd28274760899'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:52.558096 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-462eb941-7d20-406a-8d2c-7a7c29ad84db admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6509a502ce1745c79a5bd28274760899'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:52.575693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 896/1821] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:45:52 2022] DELETE /identity/v3/users/6509a502ce1745c79a5bd28274760899 => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:45:52.576807 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db8d80b-00bf-4ae4-af03-50de1886f5b8 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:52.577230 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db8d80b-00bf-4ae4-af03-50de1886f5b8 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:52.577391 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db8d80b-00bf-4ae4-af03-50de1886f5b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:52.597119 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-6db8d80b-00bf-4ae4-af03-50de1886f5b8 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:52.598337 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-6db8d80b-00bf-4ae4-af03-50de1886f5b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:52.661296 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 926/1822] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:45:52.671994 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462eb941-7d20-406a-8d2c-7a7c29ad84db admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:52.673401 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-462eb941-7d20-406a-8d2c-7a7c29ad84db admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:52.741335 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462eb941-7d20-406a-8d2c-7a7c29ad84db admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:52.746042 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a01af8-8533-413b-af0f-4356f7efac82 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:52.746765 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a01af8-8533-413b-af0f-4356f7efac82 service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:52.747402 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a01af8-8533-413b-af0f-4356f7efac82 service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:52.747917 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4a01af8-8533-413b-af0f-4356f7efac82 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:52.760663 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4a01af8-8533-413b-af0f-4356f7efac82 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:52.762373 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4a01af8-8533-413b-af0f-4356f7efac82 service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:52.773662 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 897/1823] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:45:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:45:53.325794 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6db8d80b-00bf-4ae4-af03-50de1886f5b8 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:53.327593 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-6db8d80b-00bf-4ae4-af03-50de1886f5b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:53.341171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6db8d80b-00bf-4ae4-af03-50de1886f5b8 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:53.345877 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139a2d89-6ed0-4a06-9c8c-453335b2256a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:53.346712 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139a2d89-6ed0-4a06-9c8c-453335b2256a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:53.346826 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139a2d89-6ed0-4a06-9c8c-453335b2256a admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:53.347097 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-139a2d89-6ed0-4a06-9c8c-453335b2256a admin admin] PATH_INFO: `/v3/projects/70c97375d3044a5090d53fa6d25bb957` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:53.348233 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-139a2d89-6ed0-4a06-9c8c-453335b2256a admin admin] RBAC: Authorizing `identity:delete_project(project_id=70c97375d3044a5090d53fa6d25bb957)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:53.350802 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-139a2d89-6ed0-4a06-9c8c-453335b2256a admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:53.370130 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:53.411385 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-139a2d89-6ed0-4a06-9c8c-453335b2256a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '70c97375d3044a5090d53fa6d25bb957', 'request_id': 'req-139a2d89-6ed0-4a06-9c8c-453335b2256a'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:53.418536 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-139a2d89-6ed0-4a06-9c8c-453335b2256a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '70c97375d3044a5090d53fa6d25bb957', 'request_id': 'req-139a2d89-6ed0-4a06-9c8c-453335b2256a'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:53.428224 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-139a2d89-6ed0-4a06-9c8c-453335b2256a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '70c97375d3044a5090d53fa6d25bb957', 'request_id': 'req-139a2d89-6ed0-4a06-9c8c-453335b2256a'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:53.441163 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-139a2d89-6ed0-4a06-9c8c-453335b2256a admin admin] The token cache is being invalidate because project 70c97375d3044a5090d53fa6d25bb957 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:45:53.441456 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-139a2d89-6ed0-4a06-9c8c-453335b2256a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:53.442315 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-139a2d89-6ed0-4a06-9c8c-453335b2256a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:53.443931 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 927/1824] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:45:53 2022] DELETE /identity/v3/projects/70c97375d3044a5090d53fa6d25bb957 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:45:53.447864 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a01af8-8533-413b-af0f-4356f7efac82 service neutron] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:53.455132 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:53.460009 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4a01af8-8533-413b-af0f-4356f7efac82 service neutron] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:53.537510 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d947975-b0b1-48a5-9953-80750f90e1c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:53.538580 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d947975-b0b1-48a5-9953-80750f90e1c5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:53.539078 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d947975-b0b1-48a5-9953-80750f90e1c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:53.539645 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d947975-b0b1-48a5-9953-80750f90e1c5 admin admin] PATH_INFO: `/v3/users/1c5a105ebf334a5b9cbf09d809db78aa` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:53.542467 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d947975-b0b1-48a5-9953-80750f90e1c5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1c5a105ebf334a5b9cbf09d809db78aa)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:53.545315 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d947975-b0b1-48a5-9953-80750f90e1c5 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:53.614213 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-3d947975-b0b1-48a5-9953-80750f90e1c5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1c5a105ebf334a5b9cbf09d809db78aa'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:53.629645 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-3d947975-b0b1-48a5-9953-80750f90e1c5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1c5a105ebf334a5b9cbf09d809db78aa'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:53.630888 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-3d947975-b0b1-48a5-9953-80750f90e1c5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1c5a105ebf334a5b9cbf09d809db78aa'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:53.632351 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-3d947975-b0b1-48a5-9953-80750f90e1c5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1c5a105ebf334a5b9cbf09d809db78aa'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:53.647011 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-3d947975-b0b1-48a5-9953-80750f90e1c5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1c5a105ebf334a5b9cbf09d809db78aa'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:53.688227 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 898/1825] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:45:53 2022] DELETE /identity/v3/users/1c5a105ebf334a5b9cbf09d809db78aa => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:45:54.014644 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139a2d89-6ed0-4a06-9c8c-453335b2256a admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:54.016193 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-139a2d89-6ed0-4a06-9c8c-453335b2256a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:54.036925 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-139a2d89-6ed0-4a06-9c8c-453335b2256a admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:54.121537 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d947975-b0b1-48a5-9953-80750f90e1c5 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:54.123380 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-3d947975-b0b1-48a5-9953-80750f90e1c5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:54.144146 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d6e6d6a-ffa3-4881-9133-f620ee833e64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:54.145059 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6e6d6a-ffa3-4881-9133-f620ee833e64 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:54.145585 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6e6d6a-ffa3-4881-9133-f620ee833e64 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:54.146080 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d6e6d6a-ffa3-4881-9133-f620ee833e64 admin admin] PATH_INFO: `/v3/users/f91fa09755ac4369abcc8a025e6525ae` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:54.147806 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d6e6d6a-ffa3-4881-9133-f620ee833e64 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f91fa09755ac4369abcc8a025e6525ae)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:54.150077 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d6e6d6a-ffa3-4881-9133-f620ee833e64 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:54.228224 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-3d6e6d6a-ffa3-4881-9133-f620ee833e64 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f91fa09755ac4369abcc8a025e6525ae'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:54.231137 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d947975-b0b1-48a5-9953-80750f90e1c5 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:54.237545 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-3d6e6d6a-ffa3-4881-9133-f620ee833e64 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f91fa09755ac4369abcc8a025e6525ae'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:54.238175 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-3d6e6d6a-ffa3-4881-9133-f620ee833e64 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f91fa09755ac4369abcc8a025e6525ae'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:54.238929 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-3d6e6d6a-ffa3-4881-9133-f620ee833e64 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f91fa09755ac4369abcc8a025e6525ae'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:54.239478 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-568f1116-94dc-4491-8f25-7b6bd6c8d5ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:54.240307 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568f1116-94dc-4491-8f25-7b6bd6c8d5ea admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:54.240677 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568f1116-94dc-4491-8f25-7b6bd6c8d5ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:54.240945 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568f1116-94dc-4491-8f25-7b6bd6c8d5ea admin admin] PATH_INFO: `/v3/projects/c495011bcbf64285a8961a5920b89934` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:54.242218 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-568f1116-94dc-4491-8f25-7b6bd6c8d5ea admin admin] RBAC: Authorizing `identity:delete_project(project_id=c495011bcbf64285a8961a5920b89934)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:54.244182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-568f1116-94dc-4491-8f25-7b6bd6c8d5ea admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:54.249147 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-3d6e6d6a-ffa3-4881-9133-f620ee833e64 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f91fa09755ac4369abcc8a025e6525ae'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:54.276317 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 928/1826] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:45:54 2022] DELETE /identity/v3/users/f91fa09755ac4369abcc8a025e6525ae => generated 0 bytes in 262 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:45:54.280975 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:54.284955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff3418e-a534-4854-8cbd-272aa564e465 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:54.285263 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff3418e-a534-4854-8cbd-272aa564e465 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:54.285553 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff3418e-a534-4854-8cbd-272aa564e465 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:54.306308 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-568f1116-94dc-4491-8f25-7b6bd6c8d5ea admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c495011bcbf64285a8961a5920b89934', 'request_id': 'req-568f1116-94dc-4491-8f25-7b6bd6c8d5ea'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:54.313585 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-5ff3418e-a534-4854-8cbd-272aa564e465 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:54.314758 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-568f1116-94dc-4491-8f25-7b6bd6c8d5ea admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c495011bcbf64285a8961a5920b89934', 'request_id': 'req-568f1116-94dc-4491-8f25-7b6bd6c8d5ea'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:54.315279 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-5ff3418e-a534-4854-8cbd-272aa564e465 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:54.325339 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-568f1116-94dc-4491-8f25-7b6bd6c8d5ea admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c495011bcbf64285a8961a5920b89934', 'request_id': 'req-568f1116-94dc-4491-8f25-7b6bd6c8d5ea'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:54.337343 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-568f1116-94dc-4491-8f25-7b6bd6c8d5ea admin admin] The token cache is being invalidate because project c495011bcbf64285a8961a5920b89934 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:45:54.337550 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-568f1116-94dc-4491-8f25-7b6bd6c8d5ea admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:54.338213 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-568f1116-94dc-4491-8f25-7b6bd6c8d5ea admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:54.339763 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 899/1827] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:45:54 2022] DELETE /identity/v3/projects/c495011bcbf64285a8961a5920b89934 => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:45:54.340932 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:54.345763 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f93f45-5f84-4f97-bae0-ba3991a31878 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:54.346059 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f93f45-5f84-4f97-bae0-ba3991a31878 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:54.346342 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f93f45-5f84-4f97-bae0-ba3991a31878 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:54.373182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-30f93f45-5f84-4f97-bae0-ba3991a31878 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:54.374526 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-30f93f45-5f84-4f97-bae0-ba3991a31878 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:54.386826 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 929/1828] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:45:54.398513 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff3418e-a534-4854-8cbd-272aa564e465 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:54.400524 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-5ff3418e-a534-4854-8cbd-272aa564e465 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:54.446821 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 900/1829] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:45:54.455164 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8cfdcd6-8109-413b-a4b8-641ea9562705 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:54.455812 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8cfdcd6-8109-413b-a4b8-641ea9562705 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:54.455812 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8cfdcd6-8109-413b-a4b8-641ea9562705 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:54.478443 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-a8cfdcd6-8109-413b-a4b8-641ea9562705 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:54.480083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-a8cfdcd6-8109-413b-a4b8-641ea9562705 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:54.481224 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff3418e-a534-4854-8cbd-272aa564e465 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:54.485921 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df68929-828f-4e7b-8c0d-d6aa4e1b76dd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:54.486745 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df68929-828f-4e7b-8c0d-d6aa4e1b76dd service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:54.487203 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df68929-828f-4e7b-8c0d-d6aa4e1b76dd service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:54.487837 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df68929-828f-4e7b-8c0d-d6aa4e1b76dd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:54.493703 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 901/1830] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:45:54.494830 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8cfdcd6-8109-413b-a4b8-641ea9562705 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:54.499928 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df68929-828f-4e7b-8c0d-d6aa4e1b76dd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:54.501658 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df68929-828f-4e7b-8c0d-d6aa4e1b76dd service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:54.504987 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8cfdcd6-8109-413b-a4b8-641ea9562705 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:54.511891 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-434db9f3-7f7a-423d-9dcd-4ef71b41e985 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:54.512521 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434db9f3-7f7a-423d-9dcd-4ef71b41e985 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:54.512805 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434db9f3-7f7a-423d-9dcd-4ef71b41e985 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:54.513573 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-434db9f3-7f7a-423d-9dcd-4ef71b41e985 admin admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:54.513748 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-434db9f3-7f7a-423d-9dcd-4ef71b41e985 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:45:54.514053 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-434db9f3-7f7a-423d-9dcd-4ef71b41e985 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:54.514520 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 930/1831] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:45:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:45:54.515767 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-434db9f3-7f7a-423d-9dcd-4ef71b41e985 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:54.529148 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 902/1832] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:45:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:45:54.533800 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df68929-828f-4e7b-8c0d-d6aa4e1b76dd service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:54.546026 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df68929-828f-4e7b-8c0d-d6aa4e1b76dd service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:54.552137 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8017bf0-abbf-46cc-a71e-d0df9b1af349 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:54.553754 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8017bf0-abbf-46cc-a71e-d0df9b1af349 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:54.554088 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8017bf0-abbf-46cc-a71e-d0df9b1af349 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:54.554490 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8017bf0-abbf-46cc-a71e-d0df9b1af349 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:54.555232 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8017bf0-abbf-46cc-a71e-d0df9b1af349 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:54.557606 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8017bf0-abbf-46cc-a71e-d0df9b1af349 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:54.605075 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 931/1833] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:45:54 2022] POST /identity/v3/projects => generated 381 bytes in 69 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:45:54.606524 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-434db9f3-7f7a-423d-9dcd-4ef71b41e985 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:54.620528 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-434db9f3-7f7a-423d-9dcd-4ef71b41e985 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:54.717032 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773c6e3f-f305-453d-9f60-8c11976305e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:54.717756 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773c6e3f-f305-453d-9f60-8c11976305e4 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:54.718050 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773c6e3f-f305-453d-9f60-8c11976305e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:54.718326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-773c6e3f-f305-453d-9f60-8c11976305e4 admin admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:54.720445 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-773c6e3f-f305-453d-9f60-8c11976305e4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:54.723442 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-773c6e3f-f305-453d-9f60-8c11976305e4 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:54.763338 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 903/1834] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:45:54 2022] POST /identity/v3/users => generated 291 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:45:54.766804 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8017bf0-abbf-46cc-a71e-d0df9b1af349 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:54.778657 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8017bf0-abbf-46cc-a71e-d0df9b1af349 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:54.784041 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa506f5-b228-4c50-a334-8a6a63ec3d56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:54.784723 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa506f5-b228-4c50-a334-8a6a63ec3d56 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:54.785091 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa506f5-b228-4c50-a334-8a6a63ec3d56 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:54.785522 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa506f5-b228-4c50-a334-8a6a63ec3d56 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:54.786234 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfa506f5-b228-4c50-a334-8a6a63ec3d56 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:54.788759 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfa506f5-b228-4c50-a334-8a6a63ec3d56 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:54.802651 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 932/1835] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:45:54 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:45:54.807224 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773c6e3f-f305-453d-9f60-8c11976305e4 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:54.833993 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-773c6e3f-f305-453d-9f60-8c11976305e4 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:54.845402 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085662d8-13e9-4170-ad32-3b9bd4894c87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:54.845402 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085662d8-13e9-4170-ad32-3b9bd4894c87 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:54.845402 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085662d8-13e9-4170-ad32-3b9bd4894c87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:54.845402 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085662d8-13e9-4170-ad32-3b9bd4894c87 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:54.845784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-085662d8-13e9-4170-ad32-3b9bd4894c87 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:54.847459 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-085662d8-13e9-4170-ad32-3b9bd4894c87 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:54.860088 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 904/1836] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:45:54 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:45:54.865143 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa506f5-b228-4c50-a334-8a6a63ec3d56 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:54.874132 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085662d8-13e9-4170-ad32-3b9bd4894c87 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:54.882599 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-085662d8-13e9-4170-ad32-3b9bd4894c87 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:54.891404 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa506f5-b228-4c50-a334-8a6a63ec3d56 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:54.915972 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085662d8-13e9-4170-ad32-3b9bd4894c87 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:54.919119 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dbb96c8-570f-47f7-9545-d2991b660685 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:54.919811 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbb96c8-570f-47f7-9545-d2991b660685 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:54.920097 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbb96c8-570f-47f7-9545-d2991b660685 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:54.920487 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbb96c8-570f-47f7-9545-d2991b660685 admin admin] PATH_INFO: `/v3/projects/a494a7278e794ae89e8824b81cebe7bf/users/e3b22be61da94ae08ff348702697196d/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:54.922216 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c9e59a-b4da-4357-92d2-04ee49f3ff82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:54.923004 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c9e59a-b4da-4357-92d2-04ee49f3ff82 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:54.923487 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c9e59a-b4da-4357-92d2-04ee49f3ff82 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:54.923864 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c9e59a-b4da-4357-92d2-04ee49f3ff82 admin admin] PATH_INFO: `/v3/projects/850015556bfb49fe8f1acef5a8eb81da` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:54.925224 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c9e59a-b4da-4357-92d2-04ee49f3ff82 admin admin] RBAC: Authorizing `identity:delete_project(project_id=850015556bfb49fe8f1acef5a8eb81da)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:54.928000 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c9e59a-b4da-4357-92d2-04ee49f3ff82 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:54.939557 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:54.947601 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dbb96c8-570f-47f7-9545-d2991b660685 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a494a7278e794ae89e8824b81cebe7bf, user_id=e3b22be61da94ae08ff348702697196d, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:54.950845 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dbb96c8-570f-47f7-9545-d2991b660685 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:54.988977 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-50c9e59a-b4da-4357-92d2-04ee49f3ff82 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '850015556bfb49fe8f1acef5a8eb81da', 'request_id': 'req-50c9e59a-b4da-4357-92d2-04ee49f3ff82'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:54.998551 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-50c9e59a-b4da-4357-92d2-04ee49f3ff82 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '850015556bfb49fe8f1acef5a8eb81da', 'request_id': 'req-50c9e59a-b4da-4357-92d2-04ee49f3ff82'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:55.006894 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 933/1837] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:45:54 2022] PUT /identity/v3/projects/a494a7278e794ae89e8824b81cebe7bf/users/e3b22be61da94ae08ff348702697196d/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:45:55.017866 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e6ff76-93af-4272-83e0-8181363d7360 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:55.018863 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e6ff76-93af-4272-83e0-8181363d7360 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:55.019626 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e6ff76-93af-4272-83e0-8181363d7360 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:55.027631 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-50c9e59a-b4da-4357-92d2-04ee49f3ff82 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '850015556bfb49fe8f1acef5a8eb81da', 'request_id': 'req-50c9e59a-b4da-4357-92d2-04ee49f3ff82'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:55.050100 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-50c9e59a-b4da-4357-92d2-04ee49f3ff82 admin admin] The token cache is being invalidate because project 850015556bfb49fe8f1acef5a8eb81da was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:45:55.050506 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-50c9e59a-b4da-4357-92d2-04ee49f3ff82 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:55.053249 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-50c9e59a-b4da-4357-92d2-04ee49f3ff82 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:55.058258 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 905/1838] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:45:54 2022] DELETE /identity/v3/projects/850015556bfb49fe8f1acef5a8eb81da => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:45:55.062075 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c9e59a-b4da-4357-92d2-04ee49f3ff82 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:55.063856 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-50c9e59a-b4da-4357-92d2-04ee49f3ff82 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:55.085620 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-67e6ff76-93af-4272-83e0-8181363d7360 None None] MFA Rules not processed for user `e3b22be61da94ae08ff348702697196d`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:55.087072 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-67e6ff76-93af-4272-83e0-8181363d7360 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:55.152558 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c9e59a-b4da-4357-92d2-04ee49f3ff82 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:55.169497 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:55.179531 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 934/1839] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:55 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:45:55.194476 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e6ff76-93af-4272-83e0-8181363d7360 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:55.196055 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-67e6ff76-93af-4272-83e0-8181363d7360 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:55.242785 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa28780e-6b65-4be6-9885-b08ac7799f35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:55.243563 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa28780e-6b65-4be6-9885-b08ac7799f35 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:55.243846 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa28780e-6b65-4be6-9885-b08ac7799f35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:55.244123 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa28780e-6b65-4be6-9885-b08ac7799f35 admin admin] PATH_INFO: `/v3/users/7b1a5fd5976f46f8ad05193af4ab879b` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:55.245313 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa28780e-6b65-4be6-9885-b08ac7799f35 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7b1a5fd5976f46f8ad05193af4ab879b)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:55.247404 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa28780e-6b65-4be6-9885-b08ac7799f35 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:55.285625 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e6ff76-93af-4272-83e0-8181363d7360 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:55.290708 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01447ff6-5ae0-4a69-8e8a-23d837e76343 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:55.291629 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01447ff6-5ae0-4a69-8e8a-23d837e76343 service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:55.292043 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01447ff6-5ae0-4a69-8e8a-23d837e76343 service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:55.292333 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01447ff6-5ae0-4a69-8e8a-23d837e76343 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:55.303048 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01447ff6-5ae0-4a69-8e8a-23d837e76343 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:55.304474 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01447ff6-5ae0-4a69-8e8a-23d837e76343 service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:55.318515 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 935/1840] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:45:55 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:45:55.325128 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:55.326969 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-fa28780e-6b65-4be6-9885-b08ac7799f35 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7b1a5fd5976f46f8ad05193af4ab879b'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:55.349844 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-fa28780e-6b65-4be6-9885-b08ac7799f35 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7b1a5fd5976f46f8ad05193af4ab879b'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:55.351781 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-fa28780e-6b65-4be6-9885-b08ac7799f35 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7b1a5fd5976f46f8ad05193af4ab879b'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:55.353415 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-fa28780e-6b65-4be6-9885-b08ac7799f35 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7b1a5fd5976f46f8ad05193af4ab879b'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:55.373497 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-fa28780e-6b65-4be6-9885-b08ac7799f35 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7b1a5fd5976f46f8ad05193af4ab879b'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:55.429016 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 906/1841] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:45:55 2022] DELETE /identity/v3/users/7b1a5fd5976f46f8ad05193af4ab879b => generated 0 bytes in 364 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:45:55.934447 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01447ff6-5ae0-4a69-8e8a-23d837e76343 service cinder] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:55.948258 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01447ff6-5ae0-4a69-8e8a-23d837e76343 service cinder] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:56.028257 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80bd17c5-742b-4bcc-ae3f-e01e7182efb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:56.029018 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80bd17c5-742b-4bcc-ae3f-e01e7182efb9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:56.029441 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80bd17c5-742b-4bcc-ae3f-e01e7182efb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:56.029790 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80bd17c5-742b-4bcc-ae3f-e01e7182efb9 admin admin] PATH_INFO: `/v3/projects/b6f3f3448a024da9b80d46c67db75454` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:56.031387 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80bd17c5-742b-4bcc-ae3f-e01e7182efb9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b6f3f3448a024da9b80d46c67db75454)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:56.035228 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80bd17c5-742b-4bcc-ae3f-e01e7182efb9 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:56.099841 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-80bd17c5-742b-4bcc-ae3f-e01e7182efb9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b6f3f3448a024da9b80d46c67db75454', 'request_id': 'req-80bd17c5-742b-4bcc-ae3f-e01e7182efb9'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:56.107033 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-80bd17c5-742b-4bcc-ae3f-e01e7182efb9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b6f3f3448a024da9b80d46c67db75454', 'request_id': 'req-80bd17c5-742b-4bcc-ae3f-e01e7182efb9'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:56.115661 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-80bd17c5-742b-4bcc-ae3f-e01e7182efb9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b6f3f3448a024da9b80d46c67db75454', 'request_id': 'req-80bd17c5-742b-4bcc-ae3f-e01e7182efb9'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:56.127545 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-80bd17c5-742b-4bcc-ae3f-e01e7182efb9 admin admin] The token cache is being invalidate because project b6f3f3448a024da9b80d46c67db75454 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:45:56.127841 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-80bd17c5-742b-4bcc-ae3f-e01e7182efb9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:56.128490 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-80bd17c5-742b-4bcc-ae3f-e01e7182efb9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:45:56.130265 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 936/1842] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:45:55 2022] DELETE /identity/v3/projects/b6f3f3448a024da9b80d46c67db75454 => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:45:56.134924 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85cca4ca-b0f1-4937-9fa0-b3cd0860baef None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:56.135367 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85cca4ca-b0f1-4937-9fa0-b3cd0860baef None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:56.135532 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85cca4ca-b0f1-4937-9fa0-b3cd0860baef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:56.168709 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-85cca4ca-b0f1-4937-9fa0-b3cd0860baef None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:56.170908 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-85cca4ca-b0f1-4937-9fa0-b3cd0860baef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:56.234703 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 907/1843] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:45:56.242513 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d83ce90-f98f-4e8a-a2a5-23f1312a3254 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:56.242810 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d83ce90-f98f-4e8a-a2a5-23f1312a3254 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:56.243092 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d83ce90-f98f-4e8a-a2a5-23f1312a3254 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:56.264487 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-6d83ce90-f98f-4e8a-a2a5-23f1312a3254 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:56.265706 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-6d83ce90-f98f-4e8a-a2a5-23f1312a3254 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:56.274128 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 937/1844] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:45:56.278986 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85cca4ca-b0f1-4937-9fa0-b3cd0860baef None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:56.289040 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85cca4ca-b0f1-4937-9fa0-b3cd0860baef None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:56.293742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8be9670-e67b-4b01-9224-bb1f2b2c9f93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:56.294337 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8be9670-e67b-4b01-9224-bb1f2b2c9f93 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:56.294644 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8be9670-e67b-4b01-9224-bb1f2b2c9f93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:56.294917 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8be9670-e67b-4b01-9224-bb1f2b2c9f93 admin admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:56.295559 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8be9670-e67b-4b01-9224-bb1f2b2c9f93 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:45:56.295859 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8be9670-e67b-4b01-9224-bb1f2b2c9f93 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:56.297443 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8be9670-e67b-4b01-9224-bb1f2b2c9f93 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:56.308674 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 908/1845] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:45:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:45:56.312897 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d83ce90-f98f-4e8a-a2a5-23f1312a3254 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:56.323183 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d83ce90-f98f-4e8a-a2a5-23f1312a3254 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:56.328232 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a93900-94eb-4bf7-8e89-e33eac09a608 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:56.328829 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a93900-94eb-4bf7-8e89-e33eac09a608 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:56.329111 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a93900-94eb-4bf7-8e89-e33eac09a608 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:56.329386 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a93900-94eb-4bf7-8e89-e33eac09a608 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:56.330032 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a93900-94eb-4bf7-8e89-e33eac09a608 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:56.331961 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a93900-94eb-4bf7-8e89-e33eac09a608 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:56.362359 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 938/1846] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:45:56 2022] POST /identity/v3/projects => generated 375 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:45:56.366950 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8be9670-e67b-4b01-9224-bb1f2b2c9f93 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:56.376905 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8be9670-e67b-4b01-9224-bb1f2b2c9f93 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:56.436741 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8066abd-1585-4cc3-a689-6ec0a27c4fd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:56.437415 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8066abd-1585-4cc3-a689-6ec0a27c4fd7 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:56.437703 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8066abd-1585-4cc3-a689-6ec0a27c4fd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:56.437978 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8066abd-1585-4cc3-a689-6ec0a27c4fd7 admin admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:56.438647 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8066abd-1585-4cc3-a689-6ec0a27c4fd7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:56.440606 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8066abd-1585-4cc3-a689-6ec0a27c4fd7 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:56.465048 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 909/1847] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:45:56 2022] POST /identity/v3/users => generated 288 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:45:56.468991 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a93900-94eb-4bf7-8e89-e33eac09a608 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:56.476609 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:56.482208 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a93900-94eb-4bf7-8e89-e33eac09a608 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:56.489200 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f34f91d-1cd6-4fe5-b655-5eb48343a612 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:56.490118 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f34f91d-1cd6-4fe5-b655-5eb48343a612 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:56.490745 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f34f91d-1cd6-4fe5-b655-5eb48343a612 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:56.491145 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f34f91d-1cd6-4fe5-b655-5eb48343a612 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:56.491989 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f34f91d-1cd6-4fe5-b655-5eb48343a612 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:56.493691 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f34f91d-1cd6-4fe5-b655-5eb48343a612 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:56.506317 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 939/1848] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:45:56 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:45:56.508299 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8066abd-1585-4cc3-a689-6ec0a27c4fd7 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:56.518613 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8066abd-1585-4cc3-a689-6ec0a27c4fd7 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:56.524539 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac72650-ca58-4638-978b-49c24feb968a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:56.525242 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac72650-ca58-4638-978b-49c24feb968a admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:56.525604 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac72650-ca58-4638-978b-49c24feb968a admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:56.526064 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac72650-ca58-4638-978b-49c24feb968a admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:56.526863 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac72650-ca58-4638-978b-49c24feb968a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:56.528539 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac72650-ca58-4638-978b-49c24feb968a admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:56.541187 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 910/1849] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:45:56 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:45:56.545730 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f34f91d-1cd6-4fe5-b655-5eb48343a612 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:56.556052 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f34f91d-1cd6-4fe5-b655-5eb48343a612 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:56.568247 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df91e42-12fc-4da0-81f2-9634f232c8ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:56.568931 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df91e42-12fc-4da0-81f2-9634f232c8ed admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:56.569359 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df91e42-12fc-4da0-81f2-9634f232c8ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:56.569709 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df91e42-12fc-4da0-81f2-9634f232c8ed admin admin] PATH_INFO: `/v3/projects/b01bb33e4c8d45c9a6f04ce2d478d33e/users/7a613647e5f4484793af0f2537b41da4/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:56.595010 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df91e42-12fc-4da0-81f2-9634f232c8ed admin admin] RBAC: Authorizing `identity:create_grant(project_id=b01bb33e4c8d45c9a6f04ce2d478d33e, user_id=7a613647e5f4484793af0f2537b41da4, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:56.595010 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df91e42-12fc-4da0-81f2-9634f232c8ed admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:56.608470 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 940/1850] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:45:56 2022] PUT /identity/v3/projects/b01bb33e4c8d45c9a6f04ce2d478d33e/users/7a613647e5f4484793af0f2537b41da4/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:45:56.619775 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2adf3ba-32c4-495c-abb4-58b4746331fe None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:56.620054 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2adf3ba-32c4-495c-abb4-58b4746331fe None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:56.620337 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2adf3ba-32c4-495c-abb4-58b4746331fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:56.650745 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-b2adf3ba-32c4-495c-abb4-58b4746331fe None None] MFA Rules not processed for user `7a613647e5f4484793af0f2537b41da4`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:45:56.651967 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-b2adf3ba-32c4-495c-abb4-58b4746331fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:56.674778 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=93,keystone:INSERT=4,keystone:DELETE=3 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:45:56.753180 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 911/1851] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:45:56 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:45:56.765026 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df91e42-12fc-4da0-81f2-9634f232c8ed admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:45:56.767079 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7df91e42-12fc-4da0-81f2-9634f232c8ed admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:45:56.835186 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df91e42-12fc-4da0-81f2-9634f232c8ed admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:45:56.840343 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d65d2d-dba3-44e4-91dc-9a3e6d67569d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:45:56.841759 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d65d2d-dba3-44e4-91dc-9a3e6d67569d service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:45:56.842052 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d65d2d-dba3-44e4-91dc-9a3e6d67569d service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:45:56.842334 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d65d2d-dba3-44e4-91dc-9a3e6d67569d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:45:56.855148 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d65d2d-dba3-44e4-91dc-9a3e6d67569d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:45:56.856583 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d65d2d-dba3-44e4-91dc-9a3e6d67569d service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:45:56.868277 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 941/1852] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:45:56 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:46:05.590667 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2adf3ba-32c4-495c-abb4-58b4746331fe None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:05.696843 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2adf3ba-32c4-495c-abb4-58b4746331fe None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:05.701894 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a223659-b18b-4623-b84a-8ceb8cd1ea2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:05.703001 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a223659-b18b-4623-b84a-8ceb8cd1ea2c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:05.703759 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a223659-b18b-4623-b84a-8ceb8cd1ea2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:05.703759 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a223659-b18b-4623-b84a-8ceb8cd1ea2c admin admin] PATH_INFO: `/v3/users/7a613647e5f4484793af0f2537b41da4` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:05.704949 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a223659-b18b-4623-b84a-8ceb8cd1ea2c admin admin] RBAC: Authorizing `identity:delete_user(user_id=7a613647e5f4484793af0f2537b41da4)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:05.706993 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a223659-b18b-4623-b84a-8ceb8cd1ea2c admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:05.769860 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-7a223659-b18b-4623-b84a-8ceb8cd1ea2c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7a613647e5f4484793af0f2537b41da4'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:05.779904 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-7a223659-b18b-4623-b84a-8ceb8cd1ea2c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7a613647e5f4484793af0f2537b41da4'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:05.780777 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-7a223659-b18b-4623-b84a-8ceb8cd1ea2c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7a613647e5f4484793af0f2537b41da4'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:05.781417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-7a223659-b18b-4623-b84a-8ceb8cd1ea2c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7a613647e5f4484793af0f2537b41da4'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:05.789923 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-7a223659-b18b-4623-b84a-8ceb8cd1ea2c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7a613647e5f4484793af0f2537b41da4'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:05.813911 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 912/1853] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:46:05 2022] DELETE /identity/v3/users/7a613647e5f4484793af0f2537b41da4 => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:05.816286 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40055ea-62d9-46dd-a789-a4dd31fbb2a0 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:05.816777 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40055ea-62d9-46dd-a789-a4dd31fbb2a0 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:05.816866 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40055ea-62d9-46dd-a789-a4dd31fbb2a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:05.850455 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-b40055ea-62d9-46dd-a789-a4dd31fbb2a0 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:05.851760 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-b40055ea-62d9-46dd-a789-a4dd31fbb2a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:05.917086 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 942/1854] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:46:05.926648 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a223659-b18b-4623-b84a-8ceb8cd1ea2c admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:05.928106 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-7a223659-b18b-4623-b84a-8ceb8cd1ea2c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:05.975765 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:05.992669 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a223659-b18b-4623-b84a-8ceb8cd1ea2c admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:05.996598 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95526e3-50d1-43fe-bb48-d93e5fd1745f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:05.997216 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95526e3-50d1-43fe-bb48-d93e5fd1745f service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:05.997497 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95526e3-50d1-43fe-bb48-d93e5fd1745f service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:05.997768 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95526e3-50d1-43fe-bb48-d93e5fd1745f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:06.007228 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95526e3-50d1-43fe-bb48-d93e5fd1745f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:06.008791 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95526e3-50d1-43fe-bb48-d93e5fd1745f service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:06.018325 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 913/1855] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:46:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:46:06.468761 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b40055ea-62d9-46dd-a789-a4dd31fbb2a0 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:06.470254 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-b40055ea-62d9-46dd-a789-a4dd31fbb2a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:06.486840 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b40055ea-62d9-46dd-a789-a4dd31fbb2a0 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:06.492538 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be3268cb-3787-4f77-9863-c64088784ff1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:06.493138 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3268cb-3787-4f77-9863-c64088784ff1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:06.493420 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3268cb-3787-4f77-9863-c64088784ff1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:06.493696 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be3268cb-3787-4f77-9863-c64088784ff1 admin admin] PATH_INFO: `/v3/projects/b01bb33e4c8d45c9a6f04ce2d478d33e` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:06.494866 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be3268cb-3787-4f77-9863-c64088784ff1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b01bb33e4c8d45c9a6f04ce2d478d33e)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:06.497117 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be3268cb-3787-4f77-9863-c64088784ff1 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:06.548572 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-be3268cb-3787-4f77-9863-c64088784ff1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b01bb33e4c8d45c9a6f04ce2d478d33e', 'request_id': 'req-be3268cb-3787-4f77-9863-c64088784ff1'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:06.559195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-be3268cb-3787-4f77-9863-c64088784ff1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b01bb33e4c8d45c9a6f04ce2d478d33e', 'request_id': 'req-be3268cb-3787-4f77-9863-c64088784ff1'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:06.569544 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-be3268cb-3787-4f77-9863-c64088784ff1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b01bb33e4c8d45c9a6f04ce2d478d33e', 'request_id': 'req-be3268cb-3787-4f77-9863-c64088784ff1'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:06.579876 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:06.590473 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-be3268cb-3787-4f77-9863-c64088784ff1 admin admin] The token cache is being invalidate because project b01bb33e4c8d45c9a6f04ce2d478d33e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:46:06.590776 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-be3268cb-3787-4f77-9863-c64088784ff1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:06.592742 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-be3268cb-3787-4f77-9863-c64088784ff1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:06.594754 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 943/1856] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:46:06 2022] DELETE /identity/v3/projects/b01bb33e4c8d45c9a6f04ce2d478d33e => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:06.602586 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b175bb5-2335-410a-a1dc-d48681aca37e None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:06.603057 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b175bb5-2335-410a-a1dc-d48681aca37e None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:06.603162 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b175bb5-2335-410a-a1dc-d48681aca37e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:06.624322 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-1b175bb5-2335-410a-a1dc-d48681aca37e None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:06.625766 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-1b175bb5-2335-410a-a1dc-d48681aca37e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:06.676456 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be3268cb-3787-4f77-9863-c64088784ff1 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:06.678540 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-be3268cb-3787-4f77-9863-c64088784ff1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:06.698326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 914/1857] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:46:06.704292 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be3268cb-3787-4f77-9863-c64088784ff1 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:06.705799 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1c0790-af2a-4c60-831c-0eafd2911a1e None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:06.706082 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1c0790-af2a-4c60-831c-0eafd2911a1e None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:06.706368 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b1c0790-af2a-4c60-831c-0eafd2911a1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:06.710848 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59603ae7-ee4c-4a09-a167-646d693ec5fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:06.711471 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59603ae7-ee4c-4a09-a167-646d693ec5fe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:06.711753 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59603ae7-ee4c-4a09-a167-646d693ec5fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:06.712031 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59603ae7-ee4c-4a09-a167-646d693ec5fe admin admin] PATH_INFO: `/v3/users/e3b22be61da94ae08ff348702697196d` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:06.713157 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59603ae7-ee4c-4a09-a167-646d693ec5fe admin admin] RBAC: Authorizing `identity:delete_user(user_id=e3b22be61da94ae08ff348702697196d)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:06.715192 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59603ae7-ee4c-4a09-a167-646d693ec5fe admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:06.726562 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-1b1c0790-af2a-4c60-831c-0eafd2911a1e None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:06.727782 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-1b1c0790-af2a-4c60-831c-0eafd2911a1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:06.735804 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 915/1858] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:46:06.740442 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1c0790-af2a-4c60-831c-0eafd2911a1e None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:06.749809 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b1c0790-af2a-4c60-831c-0eafd2911a1e None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:06.754550 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e1f37d-07ae-4984-8a23-3f35b7343e1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:06.755712 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e1f37d-07ae-4984-8a23-3f35b7343e1a admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:06.755996 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e1f37d-07ae-4984-8a23-3f35b7343e1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:06.756268 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e1f37d-07ae-4984-8a23-3f35b7343e1a admin admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:06.756906 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e1f37d-07ae-4984-8a23-3f35b7343e1a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:46:06.757209 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e1f37d-07ae-4984-8a23-3f35b7343e1a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:06.758835 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20e1f37d-07ae-4984-8a23-3f35b7343e1a admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:06.766844 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-59603ae7-ee4c-4a09-a167-646d693ec5fe admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e3b22be61da94ae08ff348702697196d'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:06.770861 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 916/1859] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:46:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:46:06.775025 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e1f37d-07ae-4984-8a23-3f35b7343e1a admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:06.775177 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-59603ae7-ee4c-4a09-a167-646d693ec5fe admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e3b22be61da94ae08ff348702697196d'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:06.775780 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-59603ae7-ee4c-4a09-a167-646d693ec5fe admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e3b22be61da94ae08ff348702697196d'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:06.776403 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-59603ae7-ee4c-4a09-a167-646d693ec5fe admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e3b22be61da94ae08ff348702697196d'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:06.784806 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-59603ae7-ee4c-4a09-a167-646d693ec5fe admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e3b22be61da94ae08ff348702697196d'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:06.784954 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e1f37d-07ae-4984-8a23-3f35b7343e1a admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:06.790225 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beae9cdd-6eac-4255-90d6-07358b32ad11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:06.790964 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beae9cdd-6eac-4255-90d6-07358b32ad11 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:06.791707 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beae9cdd-6eac-4255-90d6-07358b32ad11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:06.791909 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beae9cdd-6eac-4255-90d6-07358b32ad11 admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:06.792666 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beae9cdd-6eac-4255-90d6-07358b32ad11 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:06.794917 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beae9cdd-6eac-4255-90d6-07358b32ad11 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:06.806533 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 944/1860] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:46:06 2022] DELETE /identity/v3/users/e3b22be61da94ae08ff348702697196d => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:46:06.811015 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f558b9d-e2a2-41da-b6e9-3fa9bc43a8d5 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:06.811308 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f558b9d-e2a2-41da-b6e9-3fa9bc43a8d5 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:06.811584 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f558b9d-e2a2-41da-b6e9-3fa9bc43a8d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:06.827860 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 917/1861] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:46:06 2022] POST /identity/v3/projects => generated 383 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:46:06.831458 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beae9cdd-6eac-4255-90d6-07358b32ad11 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:06.831687 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-3f558b9d-e2a2-41da-b6e9-3fa9bc43a8d5 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:06.832877 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-3f558b9d-e2a2-41da-b6e9-3fa9bc43a8d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:06.834565 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-beae9cdd-6eac-4255-90d6-07358b32ad11 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:06.913208 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beae9cdd-6eac-4255-90d6-07358b32ad11 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:06.918560 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fde0b1-81d8-408c-9db3-69c386893a7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:06.919188 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fde0b1-81d8-408c-9db3-69c386893a7c admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:06.919439 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fde0b1-81d8-408c-9db3-69c386893a7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:06.919717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fde0b1-81d8-408c-9db3-69c386893a7c admin admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:06.920352 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62fde0b1-81d8-408c-9db3-69c386893a7c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:06.922259 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62fde0b1-81d8-408c-9db3-69c386893a7c admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:06.923053 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 945/1862] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:46:06.936288 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f558b9d-e2a2-41da-b6e9-3fa9bc43a8d5 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:06.939009 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-3f558b9d-e2a2-41da-b6e9-3fa9bc43a8d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:06.952619 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 918/1863] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:46:06 2022] POST /identity/v3/users => generated 292 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:46:06.957645 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fde0b1-81d8-408c-9db3-69c386893a7c admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:06.970627 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fde0b1-81d8-408c-9db3-69c386893a7c admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:06.976155 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ea3ae3-5164-4948-acf4-79a21d8e43d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:06.977282 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ea3ae3-5164-4948-acf4-79a21d8e43d6 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:06.977672 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ea3ae3-5164-4948-acf4-79a21d8e43d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:06.978574 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ea3ae3-5164-4948-acf4-79a21d8e43d6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:06.979334 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8ea3ae3-5164-4948-acf4-79a21d8e43d6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:06.981124 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8ea3ae3-5164-4948-acf4-79a21d8e43d6 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:06.993857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 919/1864] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:46:06 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:46:07.002045 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ea3ae3-5164-4948-acf4-79a21d8e43d6 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:07.014418 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ea3ae3-5164-4948-acf4-79a21d8e43d6 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:07.020205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb3f6fc-1281-46e7-8d87-93da2455f977 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:07.021477 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb3f6fc-1281-46e7-8d87-93da2455f977 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:07.021840 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb3f6fc-1281-46e7-8d87-93da2455f977 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:07.022189 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb3f6fc-1281-46e7-8d87-93da2455f977 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:07.026291 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb3f6fc-1281-46e7-8d87-93da2455f977 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:07.031151 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb3f6fc-1281-46e7-8d87-93da2455f977 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:07.043229 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f558b9d-e2a2-41da-b6e9-3fa9bc43a8d5 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:07.055771 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 920/1865] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:46:06 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:46:07.061218 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5779b306-09b1-4b50-8f68-85b37c8acc45 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:07.062233 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb3f6fc-1281-46e7-8d87-93da2455f977 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:07.065399 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5779b306-09b1-4b50-8f68-85b37c8acc45 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:07.065728 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5779b306-09b1-4b50-8f68-85b37c8acc45 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:07.066010 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5779b306-09b1-4b50-8f68-85b37c8acc45 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:07.073522 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb3f6fc-1281-46e7-8d87-93da2455f977 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:07.078304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864a0eea-a09e-4861-b85f-e8e07711bb86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:07.079118 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864a0eea-a09e-4861-b85f-e8e07711bb86 admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:07.079395 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864a0eea-a09e-4861-b85f-e8e07711bb86 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:07.080077 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864a0eea-a09e-4861-b85f-e8e07711bb86 admin admin] PATH_INFO: `/v3/projects/6f99f17a6cab46039419eb3b44bdd142/users/23acf07cede847a69ca839ee200accd3/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:07.082205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:07.084437 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5779b306-09b1-4b50-8f68-85b37c8acc45 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:07.086061 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5779b306-09b1-4b50-8f68-85b37c8acc45 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:07.107405 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864a0eea-a09e-4861-b85f-e8e07711bb86 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6f99f17a6cab46039419eb3b44bdd142, user_id=23acf07cede847a69ca839ee200accd3, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:07.109230 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 946/1866] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:46:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:46:07.111701 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864a0eea-a09e-4861-b85f-e8e07711bb86 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:07.130820 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 921/1867] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:46:07 2022] PUT /identity/v3/projects/6f99f17a6cab46039419eb3b44bdd142/users/23acf07cede847a69ca839ee200accd3/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:07.139184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8deed5-1d88-4570-90e9-b4cd1b058189 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:07.139952 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8deed5-1d88-4570-90e9-b4cd1b058189 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:07.139952 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8deed5-1d88-4570-90e9-b4cd1b058189 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:07.163994 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-fb8deed5-1d88-4570-90e9-b4cd1b058189 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:07.165334 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-fb8deed5-1d88-4570-90e9-b4cd1b058189 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:07.172234 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:07.272311 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 947/1868] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:46:07.287182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864a0eea-a09e-4861-b85f-e8e07711bb86 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:07.362649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864a0eea-a09e-4861-b85f-e8e07711bb86 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:07.367112 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8eef4a6-00e9-4d5d-98aa-9ddf6885534d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:07.367855 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8eef4a6-00e9-4d5d-98aa-9ddf6885534d service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:07.368285 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8eef4a6-00e9-4d5d-98aa-9ddf6885534d service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:07.368653 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8eef4a6-00e9-4d5d-98aa-9ddf6885534d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:07.379289 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8eef4a6-00e9-4d5d-98aa-9ddf6885534d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:07.381310 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8eef4a6-00e9-4d5d-98aa-9ddf6885534d service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:07.391599 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 922/1869] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:46:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:46:07.446363 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb8deed5-1d88-4570-90e9-b4cd1b058189 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:07.448423 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-fb8deed5-1d88-4570-90e9-b4cd1b058189 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:07.464691 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb8deed5-1d88-4570-90e9-b4cd1b058189 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:07.471243 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e289cc9c-d6b7-4d2b-81f1-2127b5d81d1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:07.472526 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e289cc9c-d6b7-4d2b-81f1-2127b5d81d1a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:07.473001 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e289cc9c-d6b7-4d2b-81f1-2127b5d81d1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:07.473731 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e289cc9c-d6b7-4d2b-81f1-2127b5d81d1a admin admin] PATH_INFO: `/v3/users/5e69e5fa2fe742899f96277df3697a79` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:07.476500 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e289cc9c-d6b7-4d2b-81f1-2127b5d81d1a admin admin] RBAC: Authorizing `identity:delete_user(user_id=5e69e5fa2fe742899f96277df3697a79)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:07.480208 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e289cc9c-d6b7-4d2b-81f1-2127b5d81d1a admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:07.551729 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e289cc9c-d6b7-4d2b-81f1-2127b5d81d1a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5e69e5fa2fe742899f96277df3697a79'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:07.570572 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8eef4a6-00e9-4d5d-98aa-9ddf6885534d service neutron] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:07.572448 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-d8eef4a6-00e9-4d5d-98aa-9ddf6885534d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:07.572566 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e289cc9c-d6b7-4d2b-81f1-2127b5d81d1a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5e69e5fa2fe742899f96277df3697a79'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:07.573183 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e289cc9c-d6b7-4d2b-81f1-2127b5d81d1a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5e69e5fa2fe742899f96277df3697a79'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:07.573932 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e289cc9c-d6b7-4d2b-81f1-2127b5d81d1a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5e69e5fa2fe742899f96277df3697a79'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:07.585167 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e289cc9c-d6b7-4d2b-81f1-2127b5d81d1a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5e69e5fa2fe742899f96277df3697a79'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:07.609094 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8eef4a6-00e9-4d5d-98aa-9ddf6885534d service neutron] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:07.613528 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 948/1870] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:46:07 2022] DELETE /identity/v3/users/5e69e5fa2fe742899f96277df3697a79 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:07.619837 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1868aa24-ed24-445c-86be-40898735eb99 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:07.620419 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1868aa24-ed24-445c-86be-40898735eb99 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:07.621097 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1868aa24-ed24-445c-86be-40898735eb99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:07.634149 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c1cc589-36e3-44af-a1ab-6fa1938a28c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:07.636426 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c1cc589-36e3-44af-a1ab-6fa1938a28c8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:07.636807 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c1cc589-36e3-44af-a1ab-6fa1938a28c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:07.638045 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c1cc589-36e3-44af-a1ab-6fa1938a28c8 admin admin] PATH_INFO: `/v3/projects/a494a7278e794ae89e8824b81cebe7bf` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:07.641460 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c1cc589-36e3-44af-a1ab-6fa1938a28c8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a494a7278e794ae89e8824b81cebe7bf)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:07.648791 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c1cc589-36e3-44af-a1ab-6fa1938a28c8 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:07.649668 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-1868aa24-ed24-445c-86be-40898735eb99 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:07.651243 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-1868aa24-ed24-445c-86be-40898735eb99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:07.732645 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-5c1cc589-36e3-44af-a1ab-6fa1938a28c8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a494a7278e794ae89e8824b81cebe7bf', 'request_id': 'req-5c1cc589-36e3-44af-a1ab-6fa1938a28c8'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:07.740658 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-5c1cc589-36e3-44af-a1ab-6fa1938a28c8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a494a7278e794ae89e8824b81cebe7bf', 'request_id': 'req-5c1cc589-36e3-44af-a1ab-6fa1938a28c8'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:07.741704 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 949/1871] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:46:07.753109 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1868aa24-ed24-445c-86be-40898735eb99 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:07.756699 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-1868aa24-ed24-445c-86be-40898735eb99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:07.766832 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-5c1cc589-36e3-44af-a1ab-6fa1938a28c8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a494a7278e794ae89e8824b81cebe7bf', 'request_id': 'req-5c1cc589-36e3-44af-a1ab-6fa1938a28c8'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:07.788098 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-5c1cc589-36e3-44af-a1ab-6fa1938a28c8 admin admin] The token cache is being invalidate because project a494a7278e794ae89e8824b81cebe7bf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:46:07.788396 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-5c1cc589-36e3-44af-a1ab-6fa1938a28c8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:07.789073 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-5c1cc589-36e3-44af-a1ab-6fa1938a28c8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:07.790830 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 923/1872] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:46:07 2022] DELETE /identity/v3/projects/a494a7278e794ae89e8824b81cebe7bf => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:07.796964 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69519a31-bb67-4bd8-a46e-1e874b535fba None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:07.797625 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:07.797964 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69519a31-bb67-4bd8-a46e-1e874b535fba None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:07.798487 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69519a31-bb67-4bd8-a46e-1e874b535fba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:07.831099 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-69519a31-bb67-4bd8-a46e-1e874b535fba None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:07.833296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-69519a31-bb67-4bd8-a46e-1e874b535fba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:07.869190 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1868aa24-ed24-445c-86be-40898735eb99 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:07.915973 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 924/1873] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:46:07.924704 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab43d69-320d-4cac-8551-d9014bc8005a None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:07.925168 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab43d69-320d-4cac-8551-d9014bc8005a None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:07.925604 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab43d69-320d-4cac-8551-d9014bc8005a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:07.973203 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-6ab43d69-320d-4cac-8551-d9014bc8005a None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:07.988008 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d164ff-a511-419d-89a0-6c1cb233dd96 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:07.990671 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d164ff-a511-419d-89a0-6c1cb233dd96 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:07.991311 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d164ff-a511-419d-89a0-6c1cb233dd96 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:07.991311 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d164ff-a511-419d-89a0-6c1cb233dd96 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:07.992242 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-6ab43d69-320d-4cac-8551-d9014bc8005a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:07.993449 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-90d164ff-a511-419d-89a0-6c1cb233dd96 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:08.010788 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d164ff-a511-419d-89a0-6c1cb233dd96 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:08.012525 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d164ff-a511-419d-89a0-6c1cb233dd96 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:08.015270 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 925/1874] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:46:08.019939 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab43d69-320d-4cac-8551-d9014bc8005a None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:08.031236 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:08.042373 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 950/1875] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:46:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 290 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:46:08.045459 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab43d69-320d-4cac-8551-d9014bc8005a None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:08.064009 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c2155f-a284-45de-998a-1f850a93aa1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:08.065468 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c2155f-a284-45de-998a-1f850a93aa1c admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:08.065949 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c2155f-a284-45de-998a-1f850a93aa1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:08.066449 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c2155f-a284-45de-998a-1f850a93aa1c admin admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:08.067308 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c2155f-a284-45de-998a-1f850a93aa1c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:46:08.068325 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c2155f-a284-45de-998a-1f850a93aa1c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:08.070376 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5c2155f-a284-45de-998a-1f850a93aa1c admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:08.085976 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 926/1876] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:46:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:46:08.092950 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d164ff-a511-419d-89a0-6c1cb233dd96 service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:08.114733 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d164ff-a511-419d-89a0-6c1cb233dd96 service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:08.121617 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ebe7e08-b91d-4bfc-8ebe-3bc02116b77e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:08.122149 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebe7e08-b91d-4bfc-8ebe-3bc02116b77e admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:08.122527 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebe7e08-b91d-4bfc-8ebe-3bc02116b77e admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:08.123120 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebe7e08-b91d-4bfc-8ebe-3bc02116b77e admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:08.123809 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ebe7e08-b91d-4bfc-8ebe-3bc02116b77e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:08.125883 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ebe7e08-b91d-4bfc-8ebe-3bc02116b77e admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:08.194060 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 951/1877] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:46:08 2022] POST /identity/v3/projects => generated 385 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:46:08.198057 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c2155f-a284-45de-998a-1f850a93aa1c admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:08.212912 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c2155f-a284-45de-998a-1f850a93aa1c admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:08.284696 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e2e5ee-63e3-4422-9c7d-e564a2470dd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:08.285563 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e2e5ee-63e3-4422-9c7d-e564a2470dd2 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:08.285892 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e2e5ee-63e3-4422-9c7d-e564a2470dd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:08.286335 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e2e5ee-63e3-4422-9c7d-e564a2470dd2 admin admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:08.287115 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e2e5ee-63e3-4422-9c7d-e564a2470dd2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:08.289269 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e2e5ee-63e3-4422-9c7d-e564a2470dd2 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:08.317331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 927/1878] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:46:08 2022] POST /identity/v3/users => generated 293 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:46:08.320245 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ebe7e08-b91d-4bfc-8ebe-3bc02116b77e admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:08.338783 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ebe7e08-b91d-4bfc-8ebe-3bc02116b77e admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:08.353756 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d9e62bf-e9ff-4030-bd49-560862336b3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:08.355931 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9e62bf-e9ff-4030-bd49-560862336b3a admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:08.356791 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9e62bf-e9ff-4030-bd49-560862336b3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:08.356791 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9e62bf-e9ff-4030-bd49-560862336b3a admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:08.357147 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d9e62bf-e9ff-4030-bd49-560862336b3a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:08.361856 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d9e62bf-e9ff-4030-bd49-560862336b3a admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:08.381892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 952/1879] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:46:08 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:46:08.382433 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e2e5ee-63e3-4422-9c7d-e564a2470dd2 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:08.394295 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e2e5ee-63e3-4422-9c7d-e564a2470dd2 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:08.401717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65eeecc8-a066-4f2c-af27-f1f8a1de4bb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:08.402417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65eeecc8-a066-4f2c-af27-f1f8a1de4bb9 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:08.402834 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65eeecc8-a066-4f2c-af27-f1f8a1de4bb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:08.403114 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65eeecc8-a066-4f2c-af27-f1f8a1de4bb9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:08.404539 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65eeecc8-a066-4f2c-af27-f1f8a1de4bb9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:08.406261 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65eeecc8-a066-4f2c-af27-f1f8a1de4bb9 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:08.433794 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 928/1880] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:46:08 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:46:08.438076 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d9e62bf-e9ff-4030-bd49-560862336b3a admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:08.452200 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d9e62bf-e9ff-4030-bd49-560862336b3a admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:08.459352 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6311df-6c5b-4e86-bd0e-e010c2c0be45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:08.462362 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6311df-6c5b-4e86-bd0e-e010c2c0be45 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:08.462985 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6311df-6c5b-4e86-bd0e-e010c2c0be45 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:08.463485 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6311df-6c5b-4e86-bd0e-e010c2c0be45 admin admin] PATH_INFO: `/v3/projects/6420fd41004d4984a3d429e21a9ebda1/users/9e8f37ef065f49dc8fcf99c2bdcd8b45/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:08.484295 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6311df-6c5b-4e86-bd0e-e010c2c0be45 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6420fd41004d4984a3d429e21a9ebda1, user_id=9e8f37ef065f49dc8fcf99c2bdcd8b45, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:08.490530 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6311df-6c5b-4e86-bd0e-e010c2c0be45 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:08.530222 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d348a0f0-05a3-4766-95df-11982a1a7521 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:08.530663 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d348a0f0-05a3-4766-95df-11982a1a7521 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:08.531052 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d348a0f0-05a3-4766-95df-11982a1a7521 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:08.535797 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 953/1881] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:46:08 2022] PUT /identity/v3/projects/6420fd41004d4984a3d429e21a9ebda1/users/9e8f37ef065f49dc8fcf99c2bdcd8b45/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 80 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:08.579030 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-d348a0f0-05a3-4766-95df-11982a1a7521 None None] MFA Rules not processed for user `9e8f37ef065f49dc8fcf99c2bdcd8b45`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:08.580618 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-d348a0f0-05a3-4766-95df-11982a1a7521 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:08.746294 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 929/1882] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:08 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 217 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:46:08.802965 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6311df-6c5b-4e86-bd0e-e010c2c0be45 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:08.805558 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ec6311df-6c5b-4e86-bd0e-e010c2c0be45 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:08.910112 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d348a0f0-05a3-4766-95df-11982a1a7521 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:08.912080 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-d348a0f0-05a3-4766-95df-11982a1a7521 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:08.913386 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6311df-6c5b-4e86-bd0e-e010c2c0be45 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:08.918082 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fae8812-5557-44a0-b424-9f154f14346a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:08.919377 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fae8812-5557-44a0-b424-9f154f14346a service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:08.919492 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fae8812-5557-44a0-b424-9f154f14346a service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:08.919801 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fae8812-5557-44a0-b424-9f154f14346a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:08.933530 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fae8812-5557-44a0-b424-9f154f14346a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:08.935092 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fae8812-5557-44a0-b424-9f154f14346a service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:08.945754 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 954/1883] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:46:08 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:46:08.995747 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:09.000853 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d348a0f0-05a3-4766-95df-11982a1a7521 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:09.011464 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801e0623-f0ac-4d26-b7d5-5fe16a9b669d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:09.011800 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801e0623-f0ac-4d26-b7d5-5fe16a9b669d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:09.012184 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801e0623-f0ac-4d26-b7d5-5fe16a9b669d admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:09.012633 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801e0623-f0ac-4d26-b7d5-5fe16a9b669d admin admin] PATH_INFO: `/v3/projects/7332b23691264d34b110a6a5955c8ba1` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:09.015099 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-801e0623-f0ac-4d26-b7d5-5fe16a9b669d admin admin] RBAC: Authorizing `identity:delete_project(project_id=7332b23691264d34b110a6a5955c8ba1)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:09.018309 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-801e0623-f0ac-4d26-b7d5-5fe16a9b669d admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:09.103789 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-801e0623-f0ac-4d26-b7d5-5fe16a9b669d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7332b23691264d34b110a6a5955c8ba1', 'request_id': 'req-801e0623-f0ac-4d26-b7d5-5fe16a9b669d'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:09.113608 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-801e0623-f0ac-4d26-b7d5-5fe16a9b669d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7332b23691264d34b110a6a5955c8ba1', 'request_id': 'req-801e0623-f0ac-4d26-b7d5-5fe16a9b669d'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:09.127253 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-801e0623-f0ac-4d26-b7d5-5fe16a9b669d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7332b23691264d34b110a6a5955c8ba1', 'request_id': 'req-801e0623-f0ac-4d26-b7d5-5fe16a9b669d'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:09.147237 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-801e0623-f0ac-4d26-b7d5-5fe16a9b669d admin admin] The token cache is being invalidate because project 7332b23691264d34b110a6a5955c8ba1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:46:09.147934 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-801e0623-f0ac-4d26-b7d5-5fe16a9b669d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:09.148914 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-801e0623-f0ac-4d26-b7d5-5fe16a9b669d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:09.153031 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 930/1884] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:46:08 2022] DELETE /identity/v3/projects/7332b23691264d34b110a6a5955c8ba1 => generated 0 bytes in 243 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:46:09.159028 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fae8812-5557-44a0-b424-9f154f14346a service cinder] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:09.184094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fae8812-5557-44a0-b424-9f154f14346a service cinder] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:09.247390 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:09.309754 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f80138-6dbc-4838-bfb9-60eddbf1d8ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:09.310442 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f80138-6dbc-4838-bfb9-60eddbf1d8ed admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:09.310919 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f80138-6dbc-4838-bfb9-60eddbf1d8ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:09.311136 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f80138-6dbc-4838-bfb9-60eddbf1d8ed admin admin] PATH_INFO: `/v3/users/0fa198d1bbe04387b2d045f741c08777` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:09.312530 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70f80138-6dbc-4838-bfb9-60eddbf1d8ed admin admin] RBAC: Authorizing `identity:delete_user(user_id=0fa198d1bbe04387b2d045f741c08777)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:09.314503 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70f80138-6dbc-4838-bfb9-60eddbf1d8ed admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:09.415011 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-70f80138-6dbc-4838-bfb9-60eddbf1d8ed admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0fa198d1bbe04387b2d045f741c08777'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:09.423263 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-70f80138-6dbc-4838-bfb9-60eddbf1d8ed admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0fa198d1bbe04387b2d045f741c08777'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:09.424208 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-70f80138-6dbc-4838-bfb9-60eddbf1d8ed admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0fa198d1bbe04387b2d045f741c08777'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:09.425853 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-70f80138-6dbc-4838-bfb9-60eddbf1d8ed admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0fa198d1bbe04387b2d045f741c08777'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:09.440732 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-70f80138-6dbc-4838-bfb9-60eddbf1d8ed admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0fa198d1bbe04387b2d045f741c08777'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:09.499169 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 955/1885] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:46:09 2022] DELETE /identity/v3/users/0fa198d1bbe04387b2d045f741c08777 => generated 0 bytes in 343 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:46:10.163874 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801e0623-f0ac-4d26-b7d5-5fe16a9b669d admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:10.168482 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-801e0623-f0ac-4d26-b7d5-5fe16a9b669d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:10.194223 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801e0623-f0ac-4d26-b7d5-5fe16a9b669d admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:10.268644 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92799b4b-395c-4649-8ab6-8fb7813de041 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:10.269291 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92799b4b-395c-4649-8ab6-8fb7813de041 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:10.269574 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92799b4b-395c-4649-8ab6-8fb7813de041 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:10.269854 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92799b4b-395c-4649-8ab6-8fb7813de041 admin admin] PATH_INFO: `/v3/projects/ee3e0a7af3f44970a619bf123fdf2465` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:10.271669 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92799b4b-395c-4649-8ab6-8fb7813de041 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ee3e0a7af3f44970a619bf123fdf2465)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:10.273597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92799b4b-395c-4649-8ab6-8fb7813de041 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:10.323978 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-92799b4b-395c-4649-8ab6-8fb7813de041 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ee3e0a7af3f44970a619bf123fdf2465', 'request_id': 'req-92799b4b-395c-4649-8ab6-8fb7813de041'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:10.330468 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-92799b4b-395c-4649-8ab6-8fb7813de041 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ee3e0a7af3f44970a619bf123fdf2465', 'request_id': 'req-92799b4b-395c-4649-8ab6-8fb7813de041'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:10.339212 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-92799b4b-395c-4649-8ab6-8fb7813de041 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ee3e0a7af3f44970a619bf123fdf2465', 'request_id': 'req-92799b4b-395c-4649-8ab6-8fb7813de041'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:10.350072 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-92799b4b-395c-4649-8ab6-8fb7813de041 admin admin] The token cache is being invalidate because project ee3e0a7af3f44970a619bf123fdf2465 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:46:10.350363 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-92799b4b-395c-4649-8ab6-8fb7813de041 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:10.351451 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-92799b4b-395c-4649-8ab6-8fb7813de041 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:10.353195 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 931/1886] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:46:10 2022] DELETE /identity/v3/projects/ee3e0a7af3f44970a619bf123fdf2465 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:10.357923 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c7e7cc-b08a-423d-83ba-a91460b465cc None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:10.358471 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c7e7cc-b08a-423d-83ba-a91460b465cc None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:10.358819 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c7e7cc-b08a-423d-83ba-a91460b465cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:10.380786 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-87c7e7cc-b08a-423d-83ba-a91460b465cc None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:10.382151 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-87c7e7cc-b08a-423d-83ba-a91460b465cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:10.459851 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 956/1887] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:46:10.467551 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f72953-c260-4be9-8b6d-b08f858cae03 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:10.467851 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f72953-c260-4be9-8b6d-b08f858cae03 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:10.468109 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f72953-c260-4be9-8b6d-b08f858cae03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:10.496925 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-d7f72953-c260-4be9-8b6d-b08f858cae03 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:10.498275 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-d7f72953-c260-4be9-8b6d-b08f858cae03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:10.516608 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 932/1888] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:46:10.521585 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c7e7cc-b08a-423d-83ba-a91460b465cc None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:10.534615 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c7e7cc-b08a-423d-83ba-a91460b465cc None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:10.543275 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-790a8e36-034d-40f0-b390-0b54ab7aa816 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:10.543667 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-790a8e36-034d-40f0-b390-0b54ab7aa816 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:10.543958 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-790a8e36-034d-40f0-b390-0b54ab7aa816 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:10.544235 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-790a8e36-034d-40f0-b390-0b54ab7aa816 admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:10.544959 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-790a8e36-034d-40f0-b390-0b54ab7aa816 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:46:10.545447 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-790a8e36-034d-40f0-b390-0b54ab7aa816 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:10.547149 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-790a8e36-034d-40f0-b390-0b54ab7aa816 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:10.562455 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 957/1889] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:46:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:46:10.566863 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f72953-c260-4be9-8b6d-b08f858cae03 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:10.584501 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f72953-c260-4be9-8b6d-b08f858cae03 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:10.591574 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47ac305-1b73-415f-91a5-e0a69c2d3ab5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:10.592709 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47ac305-1b73-415f-91a5-e0a69c2d3ab5 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:10.593288 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47ac305-1b73-415f-91a5-e0a69c2d3ab5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:10.594169 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47ac305-1b73-415f-91a5-e0a69c2d3ab5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:10.595106 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47ac305-1b73-415f-91a5-e0a69c2d3ab5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:10.597807 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47ac305-1b73-415f-91a5-e0a69c2d3ab5 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:10.611768 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:10.639554 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 933/1890] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:46:10 2022] POST /identity/v3/projects => generated 387 bytes in 73 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:46:10.643451 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-790a8e36-034d-40f0-b390-0b54ab7aa816 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:10.654466 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-790a8e36-034d-40f0-b390-0b54ab7aa816 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:10.703313 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:10.742627 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c82ffe75-569d-4992-a2db-d1b1d352ef09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:10.743260 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82ffe75-569d-4992-a2db-d1b1d352ef09 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:10.743544 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82ffe75-569d-4992-a2db-d1b1d352ef09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:10.743958 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82ffe75-569d-4992-a2db-d1b1d352ef09 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:10.744771 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c82ffe75-569d-4992-a2db-d1b1d352ef09 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:10.746946 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c82ffe75-569d-4992-a2db-d1b1d352ef09 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:10.787180 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 958/1891] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:46:10 2022] POST /identity/v3/users => generated 294 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:46:10.792655 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47ac305-1b73-415f-91a5-e0a69c2d3ab5 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:10.804531 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47ac305-1b73-415f-91a5-e0a69c2d3ab5 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:10.809617 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f4b3862-d8f0-4050-a376-41813d8056a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:10.810318 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4b3862-d8f0-4050-a376-41813d8056a8 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:10.810743 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4b3862-d8f0-4050-a376-41813d8056a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:10.811133 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4b3862-d8f0-4050-a376-41813d8056a8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:10.812255 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f4b3862-d8f0-4050-a376-41813d8056a8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:10.814616 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f4b3862-d8f0-4050-a376-41813d8056a8 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:10.831288 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c82ffe75-569d-4992-a2db-d1b1d352ef09 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:10.832771 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 934/1892] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:46:10 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:46:10.852913 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c82ffe75-569d-4992-a2db-d1b1d352ef09 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:10.878422 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f80fd49-8f25-4b53-82c3-28a601d488c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:10.879073 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f80fd49-8f25-4b53-82c3-28a601d488c0 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:10.879355 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f80fd49-8f25-4b53-82c3-28a601d488c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:10.879637 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f80fd49-8f25-4b53-82c3-28a601d488c0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:10.880868 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f80fd49-8f25-4b53-82c3-28a601d488c0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:10.882502 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f80fd49-8f25-4b53-82c3-28a601d488c0 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:10.910940 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 959/1893] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:46:10 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:46:10.915718 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f4b3862-d8f0-4050-a376-41813d8056a8 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:10.927746 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f4b3862-d8f0-4050-a376-41813d8056a8 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:10.933194 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5734ebea-8da0-47ad-854a-cf9656d96b38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:10.934511 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5734ebea-8da0-47ad-854a-cf9656d96b38 admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:10.934798 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5734ebea-8da0-47ad-854a-cf9656d96b38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:10.935184 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5734ebea-8da0-47ad-854a-cf9656d96b38 admin admin] PATH_INFO: `/v3/projects/cf55b636326949af89a39bdc5087e60e/users/155695ddd4a547609a48a8769bcb7f9c/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:10.951226 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5734ebea-8da0-47ad-854a-cf9656d96b38 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cf55b636326949af89a39bdc5087e60e, user_id=155695ddd4a547609a48a8769bcb7f9c, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:10.954019 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5734ebea-8da0-47ad-854a-cf9656d96b38 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:10.978008 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 935/1894] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:46:10 2022] PUT /identity/v3/projects/cf55b636326949af89a39bdc5087e60e/users/155695ddd4a547609a48a8769bcb7f9c/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:10.988790 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab8a09b-6f15-4f66-ad7f-9dde0a4e2554 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:10.989187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab8a09b-6f15-4f66-ad7f-9dde0a4e2554 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:10.989453 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab8a09b-6f15-4f66-ad7f-9dde0a4e2554 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:11.056390 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-eab8a09b-6f15-4f66-ad7f-9dde0a4e2554 None None] MFA Rules not processed for user `155695ddd4a547609a48a8769bcb7f9c`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:11.057930 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-eab8a09b-6f15-4f66-ad7f-9dde0a4e2554 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:11.173752 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 960/1895] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:10 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:46:11.178641 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5734ebea-8da0-47ad-854a-cf9656d96b38 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:11.278213 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5734ebea-8da0-47ad-854a-cf9656d96b38 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:11.284249 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9faee379-6b38-4830-a6aa-5671f1457e7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:11.284882 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9faee379-6b38-4830-a6aa-5671f1457e7b admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:11.285168 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9faee379-6b38-4830-a6aa-5671f1457e7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:11.285434 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9faee379-6b38-4830-a6aa-5671f1457e7b admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:11.286091 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9faee379-6b38-4830-a6aa-5671f1457e7b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:11.288288 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9faee379-6b38-4830-a6aa-5671f1457e7b admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:11.339552 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 936/1896] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:46:11 2022] POST /identity/v3/projects => generated 385 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:46:11.341578 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab8a09b-6f15-4f66-ad7f-9dde0a4e2554 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:11.356173 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab8a09b-6f15-4f66-ad7f-9dde0a4e2554 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:11.426749 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da05ebd-72c9-4fb2-b9f6-c12fce14013a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:11.427997 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da05ebd-72c9-4fb2-b9f6-c12fce14013a admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:11.428307 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da05ebd-72c9-4fb2-b9f6-c12fce14013a admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:11.428585 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da05ebd-72c9-4fb2-b9f6-c12fce14013a admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:11.429221 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da05ebd-72c9-4fb2-b9f6-c12fce14013a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:11.431281 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da05ebd-72c9-4fb2-b9f6-c12fce14013a admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:11.458594 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 961/1897] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:46:11 2022] POST /identity/v3/users => generated 299 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:46:11.460497 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9faee379-6b38-4830-a6aa-5671f1457e7b admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:11.473936 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9faee379-6b38-4830-a6aa-5671f1457e7b admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:11.479721 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd892d4-7007-46e8-abaf-f9cd99895533 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:11.480423 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd892d4-7007-46e8-abaf-f9cd99895533 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:11.480774 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd892d4-7007-46e8-abaf-f9cd99895533 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:11.481278 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd892d4-7007-46e8-abaf-f9cd99895533 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:11.482059 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd892d4-7007-46e8-abaf-f9cd99895533 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:11.483885 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd892d4-7007-46e8-abaf-f9cd99895533 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:11.500381 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 937/1898] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:46:11 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:46:11.501629 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da05ebd-72c9-4fb2-b9f6-c12fce14013a admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:11.513912 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da05ebd-72c9-4fb2-b9f6-c12fce14013a admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:11.519289 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3155a41-61b1-4dd6-b1b5-c80c7b010696 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:11.519963 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3155a41-61b1-4dd6-b1b5-c80c7b010696 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:11.520265 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3155a41-61b1-4dd6-b1b5-c80c7b010696 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:11.521049 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3155a41-61b1-4dd6-b1b5-c80c7b010696 admin admin] PATH_INFO: `/v3/projects/da3a54eed2ae4badb177db372e0dfe75/users/11d119249ac04212824f6f0b27de76e1/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:11.539462 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3155a41-61b1-4dd6-b1b5-c80c7b010696 admin admin] RBAC: Authorizing `identity:create_grant(project_id=da3a54eed2ae4badb177db372e0dfe75, user_id=11d119249ac04212824f6f0b27de76e1, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:11.543736 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3155a41-61b1-4dd6-b1b5-c80c7b010696 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:11.566362 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 962/1899] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:46:11 2022] PUT /identity/v3/projects/da3a54eed2ae4badb177db372e0dfe75/users/11d119249ac04212824f6f0b27de76e1/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:46:11.572197 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd892d4-7007-46e8-abaf-f9cd99895533 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:11.585418 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd892d4-7007-46e8-abaf-f9cd99895533 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:11.652797 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-734981df-0abe-408a-8ef7-215600bd5db3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:11.653620 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734981df-0abe-408a-8ef7-215600bd5db3 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:11.654372 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734981df-0abe-408a-8ef7-215600bd5db3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:11.654372 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734981df-0abe-408a-8ef7-215600bd5db3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:11.655174 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-734981df-0abe-408a-8ef7-215600bd5db3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:11.657339 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-734981df-0abe-408a-8ef7-215600bd5db3 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:11.676302 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 938/1900] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:46:11 2022] GET /identity/v3/roles => generated 1420 bytes in 103 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:46:11.677740 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3155a41-61b1-4dd6-b1b5-c80c7b010696 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:11.690685 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3155a41-61b1-4dd6-b1b5-c80c7b010696 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:11.696388 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718d7eb5-58e7-4312-af01-6484035ae763 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:11.697068 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718d7eb5-58e7-4312-af01-6484035ae763 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:11.698039 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718d7eb5-58e7-4312-af01-6484035ae763 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:11.698337 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718d7eb5-58e7-4312-af01-6484035ae763 admin admin] PATH_INFO: `/v3/projects/da3a54eed2ae4badb177db372e0dfe75/users/11d119249ac04212824f6f0b27de76e1/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:11.700497 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718d7eb5-58e7-4312-af01-6484035ae763 admin admin] RBAC: Authorizing `identity:create_grant(project_id=da3a54eed2ae4badb177db372e0dfe75, user_id=11d119249ac04212824f6f0b27de76e1, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:11.703806 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718d7eb5-58e7-4312-af01-6484035ae763 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:11.707502 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:11.721362 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 963/1901] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:46:11 2022] PUT /identity/v3/projects/da3a54eed2ae4badb177db372e0dfe75/users/11d119249ac04212824f6f0b27de76e1/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 44 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:11.733132 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d4f420-24b2-4499-a2cb-d83a17ea63ff None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:11.733655 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d4f420-24b2-4499-a2cb-d83a17ea63ff None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:11.734078 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d4f420-24b2-4499-a2cb-d83a17ea63ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:11.832584 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718d7eb5-58e7-4312-af01-6484035ae763 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:11.834749 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-718d7eb5-58e7-4312-af01-6484035ae763 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:11.846118 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-58d4f420-24b2-4499-a2cb-d83a17ea63ff None None] MFA Rules not processed for user `11d119249ac04212824f6f0b27de76e1`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:11.849285 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-58d4f420-24b2-4499-a2cb-d83a17ea63ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:11.870507 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:11.972886 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 939/1902] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:11 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 237 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:46:11.982461 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d4f420-24b2-4499-a2cb-d83a17ea63ff None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:11.983997 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-58d4f420-24b2-4499-a2cb-d83a17ea63ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:11.997895 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718d7eb5-58e7-4312-af01-6484035ae763 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:12.009821 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4542d76-cfcf-47e4-9935-3bd75dced86c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:12.012583 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4542d76-cfcf-47e4-9935-3bd75dced86c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:12.012853 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4542d76-cfcf-47e4-9935-3bd75dced86c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:12.015131 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4542d76-cfcf-47e4-9935-3bd75dced86c admin admin] PATH_INFO: `/v3/users/9e8f37ef065f49dc8fcf99c2bdcd8b45` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:12.017760 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4542d76-cfcf-47e4-9935-3bd75dced86c admin admin] RBAC: Authorizing `identity:delete_user(user_id=9e8f37ef065f49dc8fcf99c2bdcd8b45)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:12.033062 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4542d76-cfcf-47e4-9935-3bd75dced86c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:12.071541 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d4f420-24b2-4499-a2cb-d83a17ea63ff None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:12.098273 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b49a8b2d-b169-4031-90bb-1edcf0193d61 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:12.100487 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49a8b2d-b169-4031-90bb-1edcf0193d61 service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:12.101161 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49a8b2d-b169-4031-90bb-1edcf0193d61 service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:12.102494 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49a8b2d-b169-4031-90bb-1edcf0193d61 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:12.120670 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e4542d76-cfcf-47e4-9935-3bd75dced86c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9e8f37ef065f49dc8fcf99c2bdcd8b45'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:12.134059 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b49a8b2d-b169-4031-90bb-1edcf0193d61 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:12.134533 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e4542d76-cfcf-47e4-9935-3bd75dced86c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9e8f37ef065f49dc8fcf99c2bdcd8b45'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:12.140579 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b49a8b2d-b169-4031-90bb-1edcf0193d61 service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:12.144062 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e4542d76-cfcf-47e4-9935-3bd75dced86c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9e8f37ef065f49dc8fcf99c2bdcd8b45'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:12.144933 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e4542d76-cfcf-47e4-9935-3bd75dced86c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9e8f37ef065f49dc8fcf99c2bdcd8b45'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:12.172661 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 940/1903] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:46:11 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 191 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:46:12.183230 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e4542d76-cfcf-47e4-9935-3bd75dced86c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9e8f37ef065f49dc8fcf99c2bdcd8b45'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:12.286480 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 964/1904] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:46:11 2022] DELETE /identity/v3/users/9e8f37ef065f49dc8fcf99c2bdcd8b45 => generated 0 bytes in 454 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:12.290308 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9472daf8-aef0-4865-9bcf-a63557754c7b None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:12.290846 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9472daf8-aef0-4865-9bcf-a63557754c7b None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:12.290962 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9472daf8-aef0-4865-9bcf-a63557754c7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:12.332544 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-9472daf8-aef0-4865-9bcf-a63557754c7b None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:12.334063 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-9472daf8-aef0-4865-9bcf-a63557754c7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:12.389006 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4542d76-cfcf-47e4-9935-3bd75dced86c admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:12.390465 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e4542d76-cfcf-47e4-9935-3bd75dced86c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:12.517056 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 941/1905] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:46:12.566914 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4542d76-cfcf-47e4-9935-3bd75dced86c admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:12.588826 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9472daf8-aef0-4865-9bcf-a63557754c7b None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:12.590372 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-9472daf8-aef0-4865-9bcf-a63557754c7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:12.593711 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d17d38-2760-4a09-9a9b-554ba02beccf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:12.594599 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d17d38-2760-4a09-9a9b-554ba02beccf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:12.594891 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d17d38-2760-4a09-9a9b-554ba02beccf admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:12.595170 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d17d38-2760-4a09-9a9b-554ba02beccf admin admin] PATH_INFO: `/v3/users/155695ddd4a547609a48a8769bcb7f9c` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:12.601469 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d17d38-2760-4a09-9a9b-554ba02beccf admin admin] RBAC: Authorizing `identity:delete_user(user_id=155695ddd4a547609a48a8769bcb7f9c)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:12.605226 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d17d38-2760-4a09-9a9b-554ba02beccf admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:12.656500 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:12.691715 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9472daf8-aef0-4865-9bcf-a63557754c7b None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:12.706829 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b178d0dd-7159-4696-9e5f-c240f3dc4d74 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:12.709244 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b178d0dd-7159-4696-9e5f-c240f3dc4d74 service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:12.709871 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b178d0dd-7159-4696-9e5f-c240f3dc4d74 service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:12.710664 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b178d0dd-7159-4696-9e5f-c240f3dc4d74 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:12.728528 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:12.730288 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-b2d17d38-2760-4a09-9a9b-554ba02beccf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '155695ddd4a547609a48a8769bcb7f9c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:12.736788 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b178d0dd-7159-4696-9e5f-c240f3dc4d74 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:12.741517 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b178d0dd-7159-4696-9e5f-c240f3dc4d74 service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:12.750087 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-b2d17d38-2760-4a09-9a9b-554ba02beccf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '155695ddd4a547609a48a8769bcb7f9c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:12.751722 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-b2d17d38-2760-4a09-9a9b-554ba02beccf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '155695ddd4a547609a48a8769bcb7f9c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:12.752778 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-b2d17d38-2760-4a09-9a9b-554ba02beccf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '155695ddd4a547609a48a8769bcb7f9c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:12.769078 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-b2d17d38-2760-4a09-9a9b-554ba02beccf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '155695ddd4a547609a48a8769bcb7f9c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:12.771483 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 942/1906] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:46:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:46:12.797207 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 965/1907] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:46:12 2022] DELETE /identity/v3/users/155695ddd4a547609a48a8769bcb7f9c => generated 0 bytes in 408 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:46:12.816950 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf8ccb7-e947-449b-9348-5788d70c753b None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:12.817237 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf8ccb7-e947-449b-9348-5788d70c753b None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:12.817997 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf8ccb7-e947-449b-9348-5788d70c753b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:12.870673 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-bdf8ccb7-e947-449b-9348-5788d70c753b None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:12.874513 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-bdf8ccb7-e947-449b-9348-5788d70c753b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:13.038270 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 943/1908] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:46:13.057113 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d17d38-2760-4a09-9a9b-554ba02beccf admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:13.058981 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-b2d17d38-2760-4a09-9a9b-554ba02beccf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:13.160601 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d17d38-2760-4a09-9a9b-554ba02beccf admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:13.170196 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76fda2a3-0e22-4d12-b94d-87d1584f4aa4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:13.172259 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fda2a3-0e22-4d12-b94d-87d1584f4aa4 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:13.172713 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fda2a3-0e22-4d12-b94d-87d1584f4aa4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:13.173565 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fda2a3-0e22-4d12-b94d-87d1584f4aa4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:13.199333 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76fda2a3-0e22-4d12-b94d-87d1584f4aa4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:13.202801 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76fda2a3-0e22-4d12-b94d-87d1584f4aa4 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:13.231157 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 966/1909] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:46:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:46:13.862320 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf8ccb7-e947-449b-9348-5788d70c753b None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:13.864288 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-bdf8ccb7-e947-449b-9348-5788d70c753b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:13.890586 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf8ccb7-e947-449b-9348-5788d70c753b None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:13.910799 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9edfa09f-b1f6-4aaf-8185-d8d51090633f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:13.914345 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9edfa09f-b1f6-4aaf-8185-d8d51090633f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:13.914651 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9edfa09f-b1f6-4aaf-8185-d8d51090633f admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:13.919184 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9edfa09f-b1f6-4aaf-8185-d8d51090633f admin admin] PATH_INFO: `/v3/projects/6420fd41004d4984a3d429e21a9ebda1` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:13.924505 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9edfa09f-b1f6-4aaf-8185-d8d51090633f admin admin] RBAC: Authorizing `identity:delete_project(project_id=6420fd41004d4984a3d429e21a9ebda1)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:13.936696 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9edfa09f-b1f6-4aaf-8185-d8d51090633f admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:13.949640 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76fda2a3-0e22-4d12-b94d-87d1584f4aa4 service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:13.951748 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-76fda2a3-0e22-4d12-b94d-87d1584f4aa4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:13.975919 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76fda2a3-0e22-4d12-b94d-87d1584f4aa4 service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:13.982379 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4519577a-3dda-4954-a73e-d88eb7ffa6f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:13.983360 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4519577a-3dda-4954-a73e-d88eb7ffa6f6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:13.983815 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4519577a-3dda-4954-a73e-d88eb7ffa6f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:13.984408 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4519577a-3dda-4954-a73e-d88eb7ffa6f6 admin admin] PATH_INFO: `/v3/projects/cf55b636326949af89a39bdc5087e60e` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:13.986998 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4519577a-3dda-4954-a73e-d88eb7ffa6f6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cf55b636326949af89a39bdc5087e60e)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:13.999185 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4519577a-3dda-4954-a73e-d88eb7ffa6f6 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:14.049286 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-9edfa09f-b1f6-4aaf-8185-d8d51090633f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6420fd41004d4984a3d429e21a9ebda1', 'request_id': 'req-9edfa09f-b1f6-4aaf-8185-d8d51090633f'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:14.061645 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-9edfa09f-b1f6-4aaf-8185-d8d51090633f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6420fd41004d4984a3d429e21a9ebda1', 'request_id': 'req-9edfa09f-b1f6-4aaf-8185-d8d51090633f'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:14.080311 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-9edfa09f-b1f6-4aaf-8185-d8d51090633f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6420fd41004d4984a3d429e21a9ebda1', 'request_id': 'req-9edfa09f-b1f6-4aaf-8185-d8d51090633f'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:14.101001 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-9edfa09f-b1f6-4aaf-8185-d8d51090633f admin admin] The token cache is being invalidate because project 6420fd41004d4984a3d429e21a9ebda1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:46:14.101299 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-9edfa09f-b1f6-4aaf-8185-d8d51090633f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:14.102231 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-9edfa09f-b1f6-4aaf-8185-d8d51090633f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:14.106366 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 944/1910] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:46:13 2022] DELETE /identity/v3/projects/6420fd41004d4984a3d429e21a9ebda1 => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:14.111865 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2289abb4-a692-4d01-a2e5-6c99f4bbcd01 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:14.112151 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2289abb4-a692-4d01-a2e5-6c99f4bbcd01 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:14.112428 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2289abb4-a692-4d01-a2e5-6c99f4bbcd01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:14.122676 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-4519577a-3dda-4954-a73e-d88eb7ffa6f6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cf55b636326949af89a39bdc5087e60e', 'request_id': 'req-4519577a-3dda-4954-a73e-d88eb7ffa6f6'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:14.135929 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-4519577a-3dda-4954-a73e-d88eb7ffa6f6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cf55b636326949af89a39bdc5087e60e', 'request_id': 'req-4519577a-3dda-4954-a73e-d88eb7ffa6f6'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:14.153268 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-4519577a-3dda-4954-a73e-d88eb7ffa6f6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cf55b636326949af89a39bdc5087e60e', 'request_id': 'req-4519577a-3dda-4954-a73e-d88eb7ffa6f6'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:14.172142 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-2289abb4-a692-4d01-a2e5-6c99f4bbcd01 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:14.173413 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-2289abb4-a692-4d01-a2e5-6c99f4bbcd01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:14.181325 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-4519577a-3dda-4954-a73e-d88eb7ffa6f6 admin admin] The token cache is being invalidate because project cf55b636326949af89a39bdc5087e60e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:46:14.182193 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-4519577a-3dda-4954-a73e-d88eb7ffa6f6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:14.183440 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-4519577a-3dda-4954-a73e-d88eb7ffa6f6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:14.191361 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 967/1911] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:46:13 2022] DELETE /identity/v3/projects/cf55b636326949af89a39bdc5087e60e => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:46:14.194175 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4519577a-3dda-4954-a73e-d88eb7ffa6f6 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:14.197469 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4519577a-3dda-4954-a73e-d88eb7ffa6f6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:14.245414 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:DELETE=16,keystone:SELECT=82,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:14.302243 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 945/1912] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:46:14.304494 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d460b16b-040d-4a68-9353-557e21883c4d None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:14.304494 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d460b16b-040d-4a68-9353-557e21883c4d None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:14.304774 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d460b16b-040d-4a68-9353-557e21883c4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:14.331837 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4519577a-3dda-4954-a73e-d88eb7ffa6f6 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:14.338085 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-d460b16b-040d-4a68-9353-557e21883c4d None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:14.339339 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-d460b16b-040d-4a68-9353-557e21883c4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:14.339632 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315e7013-f9ac-433b-8799-062048dd2b5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:14.340620 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315e7013-f9ac-433b-8799-062048dd2b5e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:14.341001 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315e7013-f9ac-433b-8799-062048dd2b5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:14.341392 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315e7013-f9ac-433b-8799-062048dd2b5e admin admin] PATH_INFO: `/v3/users/11d119249ac04212824f6f0b27de76e1` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:14.342798 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-315e7013-f9ac-433b-8799-062048dd2b5e admin admin] RBAC: Authorizing `identity:delete_user(user_id=11d119249ac04212824f6f0b27de76e1)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:14.349590 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-315e7013-f9ac-433b-8799-062048dd2b5e admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:14.366146 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 946/1913] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 63 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:46:14.370480 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d460b16b-040d-4a68-9353-557e21883c4d None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:14.384503 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d460b16b-040d-4a68-9353-557e21883c4d None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:14.398560 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbee0694-50b4-40ba-aa8b-aa34b7496cdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:14.399268 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbee0694-50b4-40ba-aa8b-aa34b7496cdc admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:14.399645 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbee0694-50b4-40ba-aa8b-aa34b7496cdc admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:14.399759 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbee0694-50b4-40ba-aa8b-aa34b7496cdc admin admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:14.400506 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbee0694-50b4-40ba-aa8b-aa34b7496cdc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:46:14.400844 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbee0694-50b4-40ba-aa8b-aa34b7496cdc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:14.406361 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbee0694-50b4-40ba-aa8b-aa34b7496cdc admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:14.425104 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-315e7013-f9ac-433b-8799-062048dd2b5e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '11d119249ac04212824f6f0b27de76e1'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:14.426439 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:14.435575 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 947/1914] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:46:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:46:14.436480 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbee0694-50b4-40ba-aa8b-aa34b7496cdc admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:14.439499 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-315e7013-f9ac-433b-8799-062048dd2b5e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '11d119249ac04212824f6f0b27de76e1'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:14.440705 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-315e7013-f9ac-433b-8799-062048dd2b5e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '11d119249ac04212824f6f0b27de76e1'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:14.446792 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-315e7013-f9ac-433b-8799-062048dd2b5e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '11d119249ac04212824f6f0b27de76e1'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:14.455446 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbee0694-50b4-40ba-aa8b-aa34b7496cdc admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:14.459410 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-315e7013-f9ac-433b-8799-062048dd2b5e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '11d119249ac04212824f6f0b27de76e1'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:14.471165 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4ebb78-ae1f-4f97-9ac1-7c6dd335f466 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:14.471803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4ebb78-ae1f-4f97-9ac1-7c6dd335f466 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:14.472103 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4ebb78-ae1f-4f97-9ac1-7c6dd335f466 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:14.472473 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4ebb78-ae1f-4f97-9ac1-7c6dd335f466 admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:14.473024 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a4ebb78-ae1f-4f97-9ac1-7c6dd335f466 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:14.475521 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a4ebb78-ae1f-4f97-9ac1-7c6dd335f466 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:14.513402 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 968/1915] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:46:14 2022] DELETE /identity/v3/users/11d119249ac04212824f6f0b27de76e1 => generated 0 bytes in 321 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:14.522291 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 948/1916] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:46:14 2022] POST /identity/v3/projects => generated 399 bytes in 86 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:46:14.526375 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315e7013-f9ac-433b-8799-062048dd2b5e admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:14.529597 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-315e7013-f9ac-433b-8799-062048dd2b5e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:14.663415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315e7013-f9ac-433b-8799-062048dd2b5e admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:14.770253 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:14.798152 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad7e1cf-dc14-4f49-a19b-603849327b99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:14.798743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad7e1cf-dc14-4f49-a19b-603849327b99 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:14.799094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad7e1cf-dc14-4f49-a19b-603849327b99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:14.799438 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad7e1cf-dc14-4f49-a19b-603849327b99 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:14.800244 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ad7e1cf-dc14-4f49-a19b-603849327b99 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:14.802998 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ad7e1cf-dc14-4f49-a19b-603849327b99 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:14.835882 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 969/1917] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:46:14 2022] POST /identity/v3/users => generated 300 bytes in 310 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:46:14.840446 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4ebb78-ae1f-4f97-9ac1-7c6dd335f466 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:14.853847 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4ebb78-ae1f-4f97-9ac1-7c6dd335f466 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:14.860325 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9db71b-0de7-48a0-b3fd-31595274d929 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:14.861165 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9db71b-0de7-48a0-b3fd-31595274d929 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:14.863377 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9db71b-0de7-48a0-b3fd-31595274d929 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:14.864236 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9db71b-0de7-48a0-b3fd-31595274d929 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:14.866108 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d9db71b-0de7-48a0-b3fd-31595274d929 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:14.868949 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d9db71b-0de7-48a0-b3fd-31595274d929 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:14.886278 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 949/1918] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:46:14 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:46:14.894349 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad7e1cf-dc14-4f49-a19b-603849327b99 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:14.916677 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad7e1cf-dc14-4f49-a19b-603849327b99 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:14.922573 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b0c4f1-254c-423d-8bfb-95c97ce07380 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:14.923276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b0c4f1-254c-423d-8bfb-95c97ce07380 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:14.923679 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b0c4f1-254c-423d-8bfb-95c97ce07380 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:14.924032 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b0c4f1-254c-423d-8bfb-95c97ce07380 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:14.924844 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b0c4f1-254c-423d-8bfb-95c97ce07380 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:14.927834 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b0c4f1-254c-423d-8bfb-95c97ce07380 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:14.946535 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9db71b-0de7-48a0-b3fd-31595274d929 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:14.957856 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 970/1919] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:46:14 2022] GET /identity/v3/roles => generated 1420 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:46:14.959797 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9db71b-0de7-48a0-b3fd-31595274d929 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:14.964719 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f30f7a1-ad1e-4099-b478-03cfa354fe3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:14.965470 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f30f7a1-ad1e-4099-b478-03cfa354fe3c admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:14.965757 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f30f7a1-ad1e-4099-b478-03cfa354fe3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:14.966038 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f30f7a1-ad1e-4099-b478-03cfa354fe3c admin admin] PATH_INFO: `/v3/projects/fd9693ee1e254baa9bd612828fdeab20/users/21fcf2b0fd3b43cdac8a682a8e8b2f87/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:14.992731 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f30f7a1-ad1e-4099-b478-03cfa354fe3c admin admin] RBAC: Authorizing `identity:create_grant(project_id=fd9693ee1e254baa9bd612828fdeab20, user_id=21fcf2b0fd3b43cdac8a682a8e8b2f87, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:14.996648 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f30f7a1-ad1e-4099-b478-03cfa354fe3c admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:15.020108 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 950/1920] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:46:14 2022] PUT /identity/v3/projects/fd9693ee1e254baa9bd612828fdeab20/users/21fcf2b0fd3b43cdac8a682a8e8b2f87/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 74 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:15.031769 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f88159-a3ca-496d-9f67-0d64808ec7e8 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:15.032067 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f88159-a3ca-496d-9f67-0d64808ec7e8 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:15.032346 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f88159-a3ca-496d-9f67-0d64808ec7e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:15.073986 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-c5f88159-a3ca-496d-9f67-0d64808ec7e8 None None] MFA Rules not processed for user `21fcf2b0fd3b43cdac8a682a8e8b2f87`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:15.075297 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-c5f88159-a3ca-496d-9f67-0d64808ec7e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:15.203517 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 971/1921] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:15 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:46:15.213995 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f30f7a1-ad1e-4099-b478-03cfa354fe3c admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:15.215893 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-5f30f7a1-ad1e-4099-b478-03cfa354fe3c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:15.232654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f88159-a3ca-496d-9f67-0d64808ec7e8 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:15.234096 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-c5f88159-a3ca-496d-9f67-0d64808ec7e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:15.336692 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f30f7a1-ad1e-4099-b478-03cfa354fe3c admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:15.347258 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ceff40-01c2-4766-b927-ddcfb45f6035 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:15.349676 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ceff40-01c2-4766-b927-ddcfb45f6035 service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:15.350239 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ceff40-01c2-4766-b927-ddcfb45f6035 service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:15.350979 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ceff40-01c2-4766-b927-ddcfb45f6035 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:15.366102 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f88159-a3ca-496d-9f67-0d64808ec7e8 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:15.389008 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3129044d-2b12-4e57-bcec-6b2c9cca2d85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:15.389675 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3129044d-2b12-4e57-bcec-6b2c9cca2d85 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:15.389954 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3129044d-2b12-4e57-bcec-6b2c9cca2d85 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:15.390229 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3129044d-2b12-4e57-bcec-6b2c9cca2d85 admin admin] PATH_INFO: `/v3/projects/da3a54eed2ae4badb177db372e0dfe75` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:15.391172 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3ceff40-01c2-4766-b927-ddcfb45f6035 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:15.391461 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3129044d-2b12-4e57-bcec-6b2c9cca2d85 admin admin] RBAC: Authorizing `identity:delete_project(project_id=da3a54eed2ae4badb177db372e0dfe75)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:15.393462 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3129044d-2b12-4e57-bcec-6b2c9cca2d85 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:15.396220 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3ceff40-01c2-4766-b927-ddcfb45f6035 service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:15.415484 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 951/1922] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:46:15 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 202 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:46:15.491992 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-3129044d-2b12-4e57-bcec-6b2c9cca2d85 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'da3a54eed2ae4badb177db372e0dfe75', 'request_id': 'req-3129044d-2b12-4e57-bcec-6b2c9cca2d85'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:15.500303 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-3129044d-2b12-4e57-bcec-6b2c9cca2d85 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'da3a54eed2ae4badb177db372e0dfe75', 'request_id': 'req-3129044d-2b12-4e57-bcec-6b2c9cca2d85'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:15.503777 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=92,keystone:INSERT=4,keystone:DELETE=4 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:15.514591 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-3129044d-2b12-4e57-bcec-6b2c9cca2d85 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'da3a54eed2ae4badb177db372e0dfe75', 'request_id': 'req-3129044d-2b12-4e57-bcec-6b2c9cca2d85'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:15.544847 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-3129044d-2b12-4e57-bcec-6b2c9cca2d85 admin admin] The token cache is being invalidate because project da3a54eed2ae4badb177db372e0dfe75 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:46:15.545148 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-3129044d-2b12-4e57-bcec-6b2c9cca2d85 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:15.545900 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-3129044d-2b12-4e57-bcec-6b2c9cca2d85 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:15.547949 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 972/1923] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:46:15 2022] DELETE /identity/v3/projects/da3a54eed2ae4badb177db372e0dfe75 => generated 0 bytes in 316 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:46:15.554238 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3b314e-90f1-421f-a8af-0a20d2cecc89 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:15.554572 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3b314e-90f1-421f-a8af-0a20d2cecc89 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:15.554866 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3b314e-90f1-421f-a8af-0a20d2cecc89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:15.614605 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-dd3b314e-90f1-421f-a8af-0a20d2cecc89 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:15.615873 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-dd3b314e-90f1-421f-a8af-0a20d2cecc89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:15.789951 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 952/1924] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 231 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:46:15.793169 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08588be5-c781-4cbd-ab72-fa251a2fb34a None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:15.793749 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08588be5-c781-4cbd-ab72-fa251a2fb34a None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:15.794498 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08588be5-c781-4cbd-ab72-fa251a2fb34a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:15.835055 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-08588be5-c781-4cbd-ab72-fa251a2fb34a None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:15.836813 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-08588be5-c781-4cbd-ab72-fa251a2fb34a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:15.854494 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 973/1925] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 62 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:46:15.854683 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd3b314e-90f1-421f-a8af-0a20d2cecc89 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:15.867601 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd3b314e-90f1-421f-a8af-0a20d2cecc89 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:15.874061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-720b4d4d-7d1f-46de-8d11-fcf33eb6d899 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:15.874907 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720b4d4d-7d1f-46de-8d11-fcf33eb6d899 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:15.875407 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720b4d4d-7d1f-46de-8d11-fcf33eb6d899 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:15.876428 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720b4d4d-7d1f-46de-8d11-fcf33eb6d899 admin admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:15.877457 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-720b4d4d-7d1f-46de-8d11-fcf33eb6d899 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:46:15.877905 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-720b4d4d-7d1f-46de-8d11-fcf33eb6d899 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:15.880266 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-720b4d4d-7d1f-46de-8d11-fcf33eb6d899 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:15.922431 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 953/1926] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:46:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:46:15.925759 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08588be5-c781-4cbd-ab72-fa251a2fb34a None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:15.943693 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08588be5-c781-4cbd-ab72-fa251a2fb34a None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:15.953955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd18316-a214-474e-82b9-0f9e1f0477f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:15.954973 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd18316-a214-474e-82b9-0f9e1f0477f3 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:15.954973 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd18316-a214-474e-82b9-0f9e1f0477f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:15.955192 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd18316-a214-474e-82b9-0f9e1f0477f3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:15.955812 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dd18316-a214-474e-82b9-0f9e1f0477f3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:15.957797 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dd18316-a214-474e-82b9-0f9e1f0477f3 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:16.008202 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 974/1927] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:46:15 2022] POST /identity/v3/projects => generated 381 bytes in 83 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:46:16.012463 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-720b4d4d-7d1f-46de-8d11-fcf33eb6d899 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:16.027949 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-720b4d4d-7d1f-46de-8d11-fcf33eb6d899 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:16.102011 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:16.160717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2da9e01-2bd5-45e3-b6bf-b17477843211 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:16.162035 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2da9e01-2bd5-45e3-b6bf-b17477843211 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:16.165426 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2da9e01-2bd5-45e3-b6bf-b17477843211 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:16.165764 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2da9e01-2bd5-45e3-b6bf-b17477843211 admin admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:16.167083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2da9e01-2bd5-45e3-b6bf-b17477843211 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:16.169266 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2da9e01-2bd5-45e3-b6bf-b17477843211 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:16.203148 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 954/1928] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:46:16 2022] POST /identity/v3/users => generated 291 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:46:16.209594 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd18316-a214-474e-82b9-0f9e1f0477f3 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:16.227915 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd18316-a214-474e-82b9-0f9e1f0477f3 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:16.239184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6efe73ac-582f-48af-a7ba-f0001c1ba395 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:16.240255 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6efe73ac-582f-48af-a7ba-f0001c1ba395 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:16.240523 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6efe73ac-582f-48af-a7ba-f0001c1ba395 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:16.240798 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6efe73ac-582f-48af-a7ba-f0001c1ba395 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:16.241449 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6efe73ac-582f-48af-a7ba-f0001c1ba395 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:16.245631 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6efe73ac-582f-48af-a7ba-f0001c1ba395 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:16.293172 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 975/1929] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:46:16 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:46:16.294606 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2da9e01-2bd5-45e3-b6bf-b17477843211 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:16.315305 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2da9e01-2bd5-45e3-b6bf-b17477843211 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:16.322193 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-694dc603-2686-4045-b73b-aed222ded560 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:16.322961 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694dc603-2686-4045-b73b-aed222ded560 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:16.323467 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694dc603-2686-4045-b73b-aed222ded560 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:16.323747 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-694dc603-2686-4045-b73b-aed222ded560 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:16.324488 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-694dc603-2686-4045-b73b-aed222ded560 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:16.326330 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-694dc603-2686-4045-b73b-aed222ded560 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:16.342416 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 955/1930] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:46:16 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:46:16.342919 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6efe73ac-582f-48af-a7ba-f0001c1ba395 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:16.359007 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6efe73ac-582f-48af-a7ba-f0001c1ba395 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:16.366745 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508a8d35-69cb-433e-bdd0-02189b634d15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:16.367677 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508a8d35-69cb-433e-bdd0-02189b634d15 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:16.368105 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508a8d35-69cb-433e-bdd0-02189b634d15 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:16.368549 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-508a8d35-69cb-433e-bdd0-02189b634d15 admin admin] PATH_INFO: `/v3/projects/c684e2c6490944f0bcdd5385e39227d5/users/bc1f2bc9140f424e8a31e1e414a10629/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:16.419178 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-508a8d35-69cb-433e-bdd0-02189b634d15 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c684e2c6490944f0bcdd5385e39227d5, user_id=bc1f2bc9140f424e8a31e1e414a10629, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:16.421909 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-508a8d35-69cb-433e-bdd0-02189b634d15 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:16.463165 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 976/1931] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:46:16 2022] PUT /identity/v3/projects/c684e2c6490944f0bcdd5385e39227d5/users/bc1f2bc9140f424e8a31e1e414a10629/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:16.463717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58403e6c-a990-42ae-b3ff-0cca6d5a2d49 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:16.464056 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58403e6c-a990-42ae-b3ff-0cca6d5a2d49 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:16.464339 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58403e6c-a990-42ae-b3ff-0cca6d5a2d49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:16.511490 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-58403e6c-a990-42ae-b3ff-0cca6d5a2d49 None None] MFA Rules not processed for user `bc1f2bc9140f424e8a31e1e414a10629`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:16.513682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-58403e6c-a990-42ae-b3ff-0cca6d5a2d49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:16.625559 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 956/1932] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:16 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:46:16.630224 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508a8d35-69cb-433e-bdd0-02189b634d15 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:16.757984 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-508a8d35-69cb-433e-bdd0-02189b634d15 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:16.773002 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3f6b3c-d082-4aea-b00e-c426dc3c7036 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:16.776382 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3f6b3c-d082-4aea-b00e-c426dc3c7036 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:16.776746 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3f6b3c-d082-4aea-b00e-c426dc3c7036 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:16.777346 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3f6b3c-d082-4aea-b00e-c426dc3c7036 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:16.778315 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac3f6b3c-d082-4aea-b00e-c426dc3c7036 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:16.785492 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac3f6b3c-d082-4aea-b00e-c426dc3c7036 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:16.850095 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 977/1933] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:46:16 2022] POST /identity/v3/projects => generated 377 bytes in 217 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:46:16.851573 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58403e6c-a990-42ae-b3ff-0cca6d5a2d49 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:16.866932 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58403e6c-a990-42ae-b3ff-0cca6d5a2d49 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:16.968494 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2027cbe3-a437-4b53-ace0-cc2365290ef1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:16.969897 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2027cbe3-a437-4b53-ace0-cc2365290ef1 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:16.970345 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2027cbe3-a437-4b53-ace0-cc2365290ef1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:16.970803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2027cbe3-a437-4b53-ace0-cc2365290ef1 admin admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:16.971635 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2027cbe3-a437-4b53-ace0-cc2365290ef1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:16.975524 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2027cbe3-a437-4b53-ace0-cc2365290ef1 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:17.014490 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 957/1934] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:46:16 2022] POST /identity/v3/users => generated 295 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:46:17.015691 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3f6b3c-d082-4aea-b00e-c426dc3c7036 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:17.033002 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3f6b3c-d082-4aea-b00e-c426dc3c7036 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:17.045507 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0847c9e0-bf90-4b22-a504-a8940e3fbbc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:17.046154 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0847c9e0-bf90-4b22-a504-a8940e3fbbc2 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:17.046461 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0847c9e0-bf90-4b22-a504-a8940e3fbbc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:17.046744 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0847c9e0-bf90-4b22-a504-a8940e3fbbc2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:17.047419 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0847c9e0-bf90-4b22-a504-a8940e3fbbc2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:17.059080 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0847c9e0-bf90-4b22-a504-a8940e3fbbc2 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:17.080755 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 978/1935] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:46:17 2022] GET /identity/v3/roles => generated 1420 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:46:17.081472 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2027cbe3-a437-4b53-ace0-cc2365290ef1 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:17.095710 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2027cbe3-a437-4b53-ace0-cc2365290ef1 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:17.112003 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01eb9474-4c8b-4aa2-8ece-3091cf862fb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:17.112494 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01eb9474-4c8b-4aa2-8ece-3091cf862fb8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:17.112831 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01eb9474-4c8b-4aa2-8ece-3091cf862fb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:17.113193 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01eb9474-4c8b-4aa2-8ece-3091cf862fb8 admin admin] PATH_INFO: `/v3/projects/55bf3167b7ee406ebbbff2d856ccfbcf/users/c02c490956a741c98cd473d86dbefc32/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:17.150506 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01eb9474-4c8b-4aa2-8ece-3091cf862fb8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=55bf3167b7ee406ebbbff2d856ccfbcf, user_id=c02c490956a741c98cd473d86dbefc32, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:17.155812 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01eb9474-4c8b-4aa2-8ece-3091cf862fb8 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:17.184973 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0847c9e0-bf90-4b22-a504-a8940e3fbbc2 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:17.186337 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 958/1936] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:46:17 2022] PUT /identity/v3/projects/55bf3167b7ee406ebbbff2d856ccfbcf/users/c02c490956a741c98cd473d86dbefc32/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:17.213573 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0847c9e0-bf90-4b22-a504-a8940e3fbbc2 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:17.339048 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:DELETE=3,keystone:SELECT=94,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:17.354643 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-047bd230-dcd4-4b22-8065-ed94519ee67e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:17.355374 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047bd230-dcd4-4b22-8065-ed94519ee67e admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:17.355756 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047bd230-dcd4-4b22-8065-ed94519ee67e admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:17.356115 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047bd230-dcd4-4b22-8065-ed94519ee67e admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:17.356845 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-047bd230-dcd4-4b22-8065-ed94519ee67e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:17.359927 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-047bd230-dcd4-4b22-8065-ed94519ee67e admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:17.380930 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 979/1937] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:46:17 2022] GET /identity/v3/roles => generated 1420 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:46:17.385126 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01eb9474-4c8b-4aa2-8ece-3091cf862fb8 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:17.397405 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01eb9474-4c8b-4aa2-8ece-3091cf862fb8 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:17.406641 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e15b46-c460-4986-ad2f-9e78cb8b7ec7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:17.407987 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e15b46-c460-4986-ad2f-9e78cb8b7ec7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:17.408818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e15b46-c460-4986-ad2f-9e78cb8b7ec7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:17.409389 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e15b46-c460-4986-ad2f-9e78cb8b7ec7 admin admin] PATH_INFO: `/v3/projects/55bf3167b7ee406ebbbff2d856ccfbcf/users/c02c490956a741c98cd473d86dbefc32/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:17.413182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e15b46-c460-4986-ad2f-9e78cb8b7ec7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=55bf3167b7ee406ebbbff2d856ccfbcf, user_id=c02c490956a741c98cd473d86dbefc32, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:17.421153 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e15b46-c460-4986-ad2f-9e78cb8b7ec7 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:17.425105 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:17.450250 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 959/1938] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:46:17 2022] PUT /identity/v3/projects/55bf3167b7ee406ebbbff2d856ccfbcf/users/c02c490956a741c98cd473d86dbefc32/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:17.464064 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e606fad9-62d2-4de7-adf7-afb5556aa4ac None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:17.464671 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e606fad9-62d2-4de7-adf7-afb5556aa4ac None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:17.465055 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e606fad9-62d2-4de7-adf7-afb5556aa4ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:17.514849 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-e606fad9-62d2-4de7-adf7-afb5556aa4ac None None] MFA Rules not processed for user `c02c490956a741c98cd473d86dbefc32`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:17.516728 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e606fad9-62d2-4de7-adf7-afb5556aa4ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:17.637639 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 980/1939] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:17 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:46:17.648508 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e15b46-c460-4986-ad2f-9e78cb8b7ec7 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:17.650253 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-c0e15b46-c460-4986-ad2f-9e78cb8b7ec7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:17.725769 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e15b46-c460-4986-ad2f-9e78cb8b7ec7 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:17.729430 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35334ef4-83de-4915-b600-f903d4d6a224 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:17.730734 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35334ef4-83de-4915-b600-f903d4d6a224 service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:17.730734 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35334ef4-83de-4915-b600-f903d4d6a224 service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:17.730734 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35334ef4-83de-4915-b600-f903d4d6a224 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:17.740489 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35334ef4-83de-4915-b600-f903d4d6a224 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:17.741941 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35334ef4-83de-4915-b600-f903d4d6a224 service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:17.752689 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 960/1940] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:46:17 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:46:18.320538 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e606fad9-62d2-4de7-adf7-afb5556aa4ac None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:18.426247 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e606fad9-62d2-4de7-adf7-afb5556aa4ac None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:18.436251 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1abaf779-dd19-49ee-a766-4aa6e91375ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:18.437728 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abaf779-dd19-49ee-a766-4aa6e91375ff admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:18.438381 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abaf779-dd19-49ee-a766-4aa6e91375ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:18.438916 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abaf779-dd19-49ee-a766-4aa6e91375ff admin admin] PATH_INFO: `/v3/users/bc1f2bc9140f424e8a31e1e414a10629` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:18.440183 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1abaf779-dd19-49ee-a766-4aa6e91375ff admin admin] RBAC: Authorizing `identity:delete_user(user_id=bc1f2bc9140f424e8a31e1e414a10629)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:18.442158 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1abaf779-dd19-49ee-a766-4aa6e91375ff admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:18.520223 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-1abaf779-dd19-49ee-a766-4aa6e91375ff admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bc1f2bc9140f424e8a31e1e414a10629'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:18.529567 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-1abaf779-dd19-49ee-a766-4aa6e91375ff admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bc1f2bc9140f424e8a31e1e414a10629'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:18.530300 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-1abaf779-dd19-49ee-a766-4aa6e91375ff admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bc1f2bc9140f424e8a31e1e414a10629'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:18.531063 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-1abaf779-dd19-49ee-a766-4aa6e91375ff admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bc1f2bc9140f424e8a31e1e414a10629'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:18.540787 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-1abaf779-dd19-49ee-a766-4aa6e91375ff admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bc1f2bc9140f424e8a31e1e414a10629'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:18.553991 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:18.567404 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 981/1941] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:46:18 2022] DELETE /identity/v3/users/bc1f2bc9140f424e8a31e1e414a10629 => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:18.571934 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5305e9-7e46-4656-af52-ae1a0ab6b6a2 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:18.572975 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5305e9-7e46-4656-af52-ae1a0ab6b6a2 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:18.573261 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5305e9-7e46-4656-af52-ae1a0ab6b6a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:18.596015 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-3c5305e9-7e46-4656-af52-ae1a0ab6b6a2 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:18.597472 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-3c5305e9-7e46-4656-af52-ae1a0ab6b6a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:18.697102 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 961/1942] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:46:18.702849 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1abaf779-dd19-49ee-a766-4aa6e91375ff admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:18.704828 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-1abaf779-dd19-49ee-a766-4aa6e91375ff admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:18.790372 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1abaf779-dd19-49ee-a766-4aa6e91375ff admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:18.794838 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07a06c6b-656f-4bf5-b5c7-a06765feb0d5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:18.795468 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a06c6b-656f-4bf5-b5c7-a06765feb0d5 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:18.795747 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a06c6b-656f-4bf5-b5c7-a06765feb0d5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:18.796019 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a06c6b-656f-4bf5-b5c7-a06765feb0d5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:18.812208 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07a06c6b-656f-4bf5-b5c7-a06765feb0d5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:18.814510 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07a06c6b-656f-4bf5-b5c7-a06765feb0d5 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:18.827120 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 982/1943] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:46:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:46:19.126233 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156ee353-44e1-4eb5-bddb-fd75ed9637ef None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:19.126767 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156ee353-44e1-4eb5-bddb-fd75ed9637ef None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:19.127301 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156ee353-44e1-4eb5-bddb-fd75ed9637ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:19.163926 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-156ee353-44e1-4eb5-bddb-fd75ed9637ef None None] MFA Rules not processed for user `23acf07cede847a69ca839ee200accd3`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:19.165320 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-156ee353-44e1-4eb5-bddb-fd75ed9637ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:19.246523 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07a06c6b-656f-4bf5-b5c7-a06765feb0d5 service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:19.248149 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-07a06c6b-656f-4bf5-b5c7-a06765feb0d5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:19.249315 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 962/1944] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:19 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:46:19.253465 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-156ee353-44e1-4eb5-bddb-fd75ed9637ef None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:19.255304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-156ee353-44e1-4eb5-bddb-fd75ed9637ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:19.268907 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07a06c6b-656f-4bf5-b5c7-a06765feb0d5 service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:19.274327 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a8eaa0-8e99-419a-b4da-c4679cbdc89e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:19.275159 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a8eaa0-8e99-419a-b4da-c4679cbdc89e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:19.275555 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a8eaa0-8e99-419a-b4da-c4679cbdc89e admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:19.275897 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a8eaa0-8e99-419a-b4da-c4679cbdc89e admin admin] PATH_INFO: `/v3/projects/c684e2c6490944f0bcdd5385e39227d5` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:19.277671 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a8eaa0-8e99-419a-b4da-c4679cbdc89e admin admin] RBAC: Authorizing `identity:delete_project(project_id=c684e2c6490944f0bcdd5385e39227d5)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:19.279844 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5a8eaa0-8e99-419a-b4da-c4679cbdc89e admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:19.284681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-156ee353-44e1-4eb5-bddb-fd75ed9637ef None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:19.291139 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0dbc7a7-5040-4ff9-8d6d-bf910e163b2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:19.292433 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0dbc7a7-5040-4ff9-8d6d-bf910e163b2a admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:19.292962 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0dbc7a7-5040-4ff9-8d6d-bf910e163b2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:19.293318 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0dbc7a7-5040-4ff9-8d6d-bf910e163b2a admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:19.294183 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0dbc7a7-5040-4ff9-8d6d-bf910e163b2a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:19.297013 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0dbc7a7-5040-4ff9-8d6d-bf910e163b2a admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:19.336201 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 963/1945] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:46:19 2022] POST /identity/v3/projects => generated 385 bytes in 80 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:46:19.339520 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0dbc7a7-5040-4ff9-8d6d-bf910e163b2a admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:19.345896 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:19.366578 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-d5a8eaa0-8e99-419a-b4da-c4679cbdc89e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c684e2c6490944f0bcdd5385e39227d5', 'request_id': 'req-d5a8eaa0-8e99-419a-b4da-c4679cbdc89e'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:19.375824 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-d5a8eaa0-8e99-419a-b4da-c4679cbdc89e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c684e2c6490944f0bcdd5385e39227d5', 'request_id': 'req-d5a8eaa0-8e99-419a-b4da-c4679cbdc89e'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:19.390285 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-d5a8eaa0-8e99-419a-b4da-c4679cbdc89e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c684e2c6490944f0bcdd5385e39227d5', 'request_id': 'req-d5a8eaa0-8e99-419a-b4da-c4679cbdc89e'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:19.415399 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-d5a8eaa0-8e99-419a-b4da-c4679cbdc89e admin admin] The token cache is being invalidate because project c684e2c6490944f0bcdd5385e39227d5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:46:19.416110 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-d5a8eaa0-8e99-419a-b4da-c4679cbdc89e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:19.417366 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-d5a8eaa0-8e99-419a-b4da-c4679cbdc89e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:19.419797 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 983/1946] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:46:19 2022] DELETE /identity/v3/projects/c684e2c6490944f0bcdd5385e39227d5 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:46:19.423623 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a8eaa0-8e99-419a-b4da-c4679cbdc89e admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:19.425844 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d5a8eaa0-8e99-419a-b4da-c4679cbdc89e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:19.441563 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0dbc7a7-5040-4ff9-8d6d-bf910e163b2a admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:19.448260 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8518e1-ceaa-41ee-babb-1c8914e83be5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:19.448896 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8518e1-ceaa-41ee-babb-1c8914e83be5 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:19.449249 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8518e1-ceaa-41ee-babb-1c8914e83be5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:19.449821 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8518e1-ceaa-41ee-babb-1c8914e83be5 admin admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:19.450506 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8518e1-ceaa-41ee-babb-1c8914e83be5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:19.458984 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8518e1-ceaa-41ee-babb-1c8914e83be5 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:19.488626 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 964/1947] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:46:19 2022] POST /identity/v3/users => generated 299 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:46:19.491236 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8518e1-ceaa-41ee-babb-1c8914e83be5 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:19.492982 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-4f8518e1-ceaa-41ee-babb-1c8914e83be5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:19.506809 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8518e1-ceaa-41ee-babb-1c8914e83be5 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:19.512677 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c0775d-44a0-4fa5-a396-700762acbe1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:19.513267 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c0775d-44a0-4fa5-a396-700762acbe1a admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:19.513649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c0775d-44a0-4fa5-a396-700762acbe1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:19.514440 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c0775d-44a0-4fa5-a396-700762acbe1a admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:19.514643 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c0775d-44a0-4fa5-a396-700762acbe1a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:19.516257 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c0775d-44a0-4fa5-a396-700762acbe1a admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:19.529551 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 965/1948] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:46:19 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:46:19.533982 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c0775d-44a0-4fa5-a396-700762acbe1a admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:19.537299 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a8eaa0-8e99-419a-b4da-c4679cbdc89e admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:19.543842 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75abb07-f669-43da-8bfd-f673a8a75dfa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:19.544507 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75abb07-f669-43da-8bfd-f673a8a75dfa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:19.544789 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75abb07-f669-43da-8bfd-f673a8a75dfa admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:19.545076 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c75abb07-f669-43da-8bfd-f673a8a75dfa admin admin] PATH_INFO: `/v3/users/c02c490956a741c98cd473d86dbefc32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:19.545909 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c0775d-44a0-4fa5-a396-700762acbe1a admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:19.546370 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c75abb07-f669-43da-8bfd-f673a8a75dfa admin admin] RBAC: Authorizing `identity:delete_user(user_id=c02c490956a741c98cd473d86dbefc32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:19.548625 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c75abb07-f669-43da-8bfd-f673a8a75dfa admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:19.551121 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bee6d37-19f0-4103-94b0-8621edbe5591 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:19.552261 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bee6d37-19f0-4103-94b0-8621edbe5591 admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:19.552707 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bee6d37-19f0-4103-94b0-8621edbe5591 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:19.553120 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bee6d37-19f0-4103-94b0-8621edbe5591 admin admin] PATH_INFO: `/v3/projects/211973c63e014864a20f470121b94a0f/users/eb94858904f14e0aa1d31a2a5f6c48b1/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:19.573904 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bee6d37-19f0-4103-94b0-8621edbe5591 admin admin] RBAC: Authorizing `identity:create_grant(project_id=211973c63e014864a20f470121b94a0f, user_id=eb94858904f14e0aa1d31a2a5f6c48b1, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:19.579420 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bee6d37-19f0-4103-94b0-8621edbe5591 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:19.587014 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:19.602251 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 966/1949] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:46:19 2022] PUT /identity/v3/projects/211973c63e014864a20f470121b94a0f/users/eb94858904f14e0aa1d31a2a5f6c48b1/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:19.607530 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bee6d37-19f0-4103-94b0-8621edbe5591 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:19.623023 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c75abb07-f669-43da-8bfd-f673a8a75dfa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c02c490956a741c98cd473d86dbefc32'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:19.632174 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c75abb07-f669-43da-8bfd-f673a8a75dfa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c02c490956a741c98cd473d86dbefc32'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:19.633996 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c75abb07-f669-43da-8bfd-f673a8a75dfa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c02c490956a741c98cd473d86dbefc32'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:19.634788 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c75abb07-f669-43da-8bfd-f673a8a75dfa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c02c490956a741c98cd473d86dbefc32'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:19.645957 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c75abb07-f669-43da-8bfd-f673a8a75dfa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c02c490956a741c98cd473d86dbefc32'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:19.670744 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 984/1950] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:46:19 2022] DELETE /identity/v3/users/c02c490956a741c98cd473d86dbefc32 => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:46:19.703926 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bee6d37-19f0-4103-94b0-8621edbe5591 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:19.805235 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a26fda5-26d0-434c-9f82-fa0f13b4986b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:19.805873 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a26fda5-26d0-434c-9f82-fa0f13b4986b admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:19.806159 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a26fda5-26d0-434c-9f82-fa0f13b4986b admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:19.806512 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a26fda5-26d0-434c-9f82-fa0f13b4986b admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:19.807174 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a26fda5-26d0-434c-9f82-fa0f13b4986b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:19.808905 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a26fda5-26d0-434c-9f82-fa0f13b4986b admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:19.822479 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 967/1951] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:46:19 2022] GET /identity/v3/roles => generated 1420 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:46:19.826646 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75abb07-f669-43da-8bfd-f673a8a75dfa admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:19.828165 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-c75abb07-f669-43da-8bfd-f673a8a75dfa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:19.842772 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c75abb07-f669-43da-8bfd-f673a8a75dfa admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:19.851458 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8327117-efec-47e7-a92f-9e13547a6e34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:19.852413 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8327117-efec-47e7-a92f-9e13547a6e34 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:19.852717 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8327117-efec-47e7-a92f-9e13547a6e34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:19.860565 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8327117-efec-47e7-a92f-9e13547a6e34 admin admin] PATH_INFO: `/v3/projects/211973c63e014864a20f470121b94a0f/users/eb94858904f14e0aa1d31a2a5f6c48b1/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:19.881188 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8327117-efec-47e7-a92f-9e13547a6e34 admin admin] RBAC: Authorizing `identity:create_grant(project_id=211973c63e014864a20f470121b94a0f, user_id=eb94858904f14e0aa1d31a2a5f6c48b1, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:19.888650 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8327117-efec-47e7-a92f-9e13547a6e34 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:19.905197 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 985/1952] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:46:19 2022] PUT /identity/v3/projects/211973c63e014864a20f470121b94a0f/users/eb94858904f14e0aa1d31a2a5f6c48b1/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 79 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:20.024661 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a26fda5-26d0-434c-9f82-fa0f13b4986b admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:20.027147 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-7a26fda5-26d0-434c-9f82-fa0f13b4986b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:20.039031 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:20.046625 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a26fda5-26d0-434c-9f82-fa0f13b4986b admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:20.129832 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17ac6470-b20e-40d7-b192-9e6f416be7e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:20.131954 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ac6470-b20e-40d7-b192-9e6f416be7e5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:20.132191 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ac6470-b20e-40d7-b192-9e6f416be7e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:20.133244 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ac6470-b20e-40d7-b192-9e6f416be7e5 admin admin] PATH_INFO: `/v3/projects/55bf3167b7ee406ebbbff2d856ccfbcf` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:20.134929 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17ac6470-b20e-40d7-b192-9e6f416be7e5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=55bf3167b7ee406ebbbff2d856ccfbcf)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:20.137988 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17ac6470-b20e-40d7-b192-9e6f416be7e5 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:20.204285 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-17ac6470-b20e-40d7-b192-9e6f416be7e5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '55bf3167b7ee406ebbbff2d856ccfbcf', 'request_id': 'req-17ac6470-b20e-40d7-b192-9e6f416be7e5'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:20.212444 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-17ac6470-b20e-40d7-b192-9e6f416be7e5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '55bf3167b7ee406ebbbff2d856ccfbcf', 'request_id': 'req-17ac6470-b20e-40d7-b192-9e6f416be7e5'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:20.223569 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-17ac6470-b20e-40d7-b192-9e6f416be7e5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '55bf3167b7ee406ebbbff2d856ccfbcf', 'request_id': 'req-17ac6470-b20e-40d7-b192-9e6f416be7e5'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:20.242963 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-17ac6470-b20e-40d7-b192-9e6f416be7e5 admin admin] The token cache is being invalidate because project 55bf3167b7ee406ebbbff2d856ccfbcf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:46:20.242963 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-17ac6470-b20e-40d7-b192-9e6f416be7e5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:20.243578 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-17ac6470-b20e-40d7-b192-9e6f416be7e5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:20.245582 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 968/1953] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:46:20 2022] DELETE /identity/v3/projects/55bf3167b7ee406ebbbff2d856ccfbcf => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:20.251309 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee4f66c-9be3-42a9-b9ba-2781052a30b2 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:20.251868 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee4f66c-9be3-42a9-b9ba-2781052a30b2 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:20.252357 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee4f66c-9be3-42a9-b9ba-2781052a30b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:20.287128 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-bee4f66c-9be3-42a9-b9ba-2781052a30b2 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:20.288446 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-bee4f66c-9be3-42a9-b9ba-2781052a30b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:20.366584 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 986/1954] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:46:20.372342 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e029c915-12c8-40b6-8c40-10037f8317f0 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:20.372625 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e029c915-12c8-40b6-8c40-10037f8317f0 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:20.372924 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e029c915-12c8-40b6-8c40-10037f8317f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:20.406245 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-e029c915-12c8-40b6-8c40-10037f8317f0 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:20.407836 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-e029c915-12c8-40b6-8c40-10037f8317f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:20.420573 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 969/1955] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:46:20.422306 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee4f66c-9be3-42a9-b9ba-2781052a30b2 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:20.434760 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee4f66c-9be3-42a9-b9ba-2781052a30b2 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:20.440015 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06d4acc-17c7-456f-8a2d-0203a938c5b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:20.440780 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06d4acc-17c7-456f-8a2d-0203a938c5b5 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:20.441063 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06d4acc-17c7-456f-8a2d-0203a938c5b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:20.441345 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06d4acc-17c7-456f-8a2d-0203a938c5b5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:20.442015 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d06d4acc-17c7-456f-8a2d-0203a938c5b5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:46:20.442354 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d06d4acc-17c7-456f-8a2d-0203a938c5b5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:20.444026 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d06d4acc-17c7-456f-8a2d-0203a938c5b5 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:20.460912 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 987/1956] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:46:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:46:20.462119 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e029c915-12c8-40b6-8c40-10037f8317f0 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:20.490538 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e029c915-12c8-40b6-8c40-10037f8317f0 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:20.496678 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a852af2a-1cfa-4338-b9ff-f0bc4ac0d9f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:20.497308 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a852af2a-1cfa-4338-b9ff-f0bc4ac0d9f0 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:20.497595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a852af2a-1cfa-4338-b9ff-f0bc4ac0d9f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:20.497870 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a852af2a-1cfa-4338-b9ff-f0bc4ac0d9f0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:20.498562 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a852af2a-1cfa-4338-b9ff-f0bc4ac0d9f0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:20.500478 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a852af2a-1cfa-4338-b9ff-f0bc4ac0d9f0 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:20.542447 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 970/1957] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:46:20 2022] POST /identity/v3/projects => generated 381 bytes in 81 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:46:20.547237 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06d4acc-17c7-456f-8a2d-0203a938c5b5 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:20.569794 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06d4acc-17c7-456f-8a2d-0203a938c5b5 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:20.659759 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767db502-3941-47e3-969c-1ae5b6437d1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:20.660326 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767db502-3941-47e3-969c-1ae5b6437d1e admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:20.660608 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767db502-3941-47e3-969c-1ae5b6437d1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:20.660893 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767db502-3941-47e3-969c-1ae5b6437d1e admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:20.662067 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-767db502-3941-47e3-969c-1ae5b6437d1e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:20.664721 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-767db502-3941-47e3-969c-1ae5b6437d1e admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:20.711405 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 988/1958] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:46:20 2022] POST /identity/v3/users => generated 291 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:46:20.715229 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a852af2a-1cfa-4338-b9ff-f0bc4ac0d9f0 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:20.728438 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a852af2a-1cfa-4338-b9ff-f0bc4ac0d9f0 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:20.733532 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b4d3a4-168f-4be5-b29a-e138d5bce5c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:20.734154 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b4d3a4-168f-4be5-b29a-e138d5bce5c7 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:20.734646 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b4d3a4-168f-4be5-b29a-e138d5bce5c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:20.734924 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b4d3a4-168f-4be5-b29a-e138d5bce5c7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:20.735574 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b4d3a4-168f-4be5-b29a-e138d5bce5c7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:20.737190 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b4d3a4-168f-4be5-b29a-e138d5bce5c7 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:20.751449 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 971/1959] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:46:20 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:46:20.755792 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767db502-3941-47e3-969c-1ae5b6437d1e admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:20.764823 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=6,keystone:INSERT=5 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:20.778143 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767db502-3941-47e3-969c-1ae5b6437d1e admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:20.792914 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-212017f8-4d23-4446-a74b-abcdd741c31c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:20.793785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212017f8-4d23-4446-a74b-abcdd741c31c admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:20.794068 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212017f8-4d23-4446-a74b-abcdd741c31c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:20.795781 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212017f8-4d23-4446-a74b-abcdd741c31c admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:20.796471 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-212017f8-4d23-4446-a74b-abcdd741c31c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:20.804480 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-212017f8-4d23-4446-a74b-abcdd741c31c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:20.835638 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 989/1960] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:46:20 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:46:20.836892 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b4d3a4-168f-4be5-b29a-e138d5bce5c7 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:20.851463 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b4d3a4-168f-4be5-b29a-e138d5bce5c7 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:20.858105 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fa8cc6-a263-4f6d-be6f-23e32b4fdb1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:20.859131 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fa8cc6-a263-4f6d-be6f-23e32b4fdb1f admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:20.859610 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fa8cc6-a263-4f6d-be6f-23e32b4fdb1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:20.860095 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fa8cc6-a263-4f6d-be6f-23e32b4fdb1f admin admin] PATH_INFO: `/v3/projects/00e9f9d45a7c46e4a65aa6afa1e49778/users/da9f6060b0e848aea19e89ebe3c9bed6/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:20.881028 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0fa8cc6-a263-4f6d-be6f-23e32b4fdb1f admin admin] RBAC: Authorizing `identity:create_grant(project_id=00e9f9d45a7c46e4a65aa6afa1e49778, user_id=da9f6060b0e848aea19e89ebe3c9bed6, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:20.884364 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0fa8cc6-a263-4f6d-be6f-23e32b4fdb1f admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:20.914788 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 972/1961] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:46:20 2022] PUT /identity/v3/projects/00e9f9d45a7c46e4a65aa6afa1e49778/users/da9f6060b0e848aea19e89ebe3c9bed6/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:20.926236 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ca22a5-237f-4f0a-bb86-3b00ce3da144 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:20.926557 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ca22a5-237f-4f0a-bb86-3b00ce3da144 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:20.927383 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ca22a5-237f-4f0a-bb86-3b00ce3da144 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:21.016151 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-70ca22a5-237f-4f0a-bb86-3b00ce3da144 None None] MFA Rules not processed for user `da9f6060b0e848aea19e89ebe3c9bed6`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:21.017430 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-70ca22a5-237f-4f0a-bb86-3b00ce3da144 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:21.175079 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 990/1962] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:20 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 249 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:46:21.178445 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fa8cc6-a263-4f6d-be6f-23e32b4fdb1f admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:21.255800 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:21.327592 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fa8cc6-a263-4f6d-be6f-23e32b4fdb1f admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:21.334916 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08553b57-3af6-4adf-a638-37c6a397a360 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:21.336489 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08553b57-3af6-4adf-a638-37c6a397a360 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:21.336989 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08553b57-3af6-4adf-a638-37c6a397a360 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:21.337529 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08553b57-3af6-4adf-a638-37c6a397a360 admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:21.338421 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08553b57-3af6-4adf-a638-37c6a397a360 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:21.341448 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08553b57-3af6-4adf-a638-37c6a397a360 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:21.389569 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 973/1963] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:46:21 2022] POST /identity/v3/projects => generated 381 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:46:21.393462 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ca22a5-237f-4f0a-bb86-3b00ce3da144 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:21.409526 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ca22a5-237f-4f0a-bb86-3b00ce3da144 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:21.551469 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4413fe-a9c6-4c2e-9f0c-c959c9de7f31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:21.552246 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4413fe-a9c6-4c2e-9f0c-c959c9de7f31 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:21.561200 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4413fe-a9c6-4c2e-9f0c-c959c9de7f31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:21.561530 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4413fe-a9c6-4c2e-9f0c-c959c9de7f31 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:21.562449 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d4413fe-a9c6-4c2e-9f0c-c959c9de7f31 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:21.566629 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d4413fe-a9c6-4c2e-9f0c-c959c9de7f31 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:21.626267 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 991/1964] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:46:21 2022] POST /identity/v3/users => generated 297 bytes in 233 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:46:21.632111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08553b57-3af6-4adf-a638-37c6a397a360 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:21.647785 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08553b57-3af6-4adf-a638-37c6a397a360 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:21.665340 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ffee83-fea3-427d-aa75-a6ca4a2587ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:21.666763 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ffee83-fea3-427d-aa75-a6ca4a2587ef admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:21.667276 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ffee83-fea3-427d-aa75-a6ca4a2587ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:21.667748 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ffee83-fea3-427d-aa75-a6ca4a2587ef admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:21.668622 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ffee83-fea3-427d-aa75-a6ca4a2587ef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:21.671571 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96ffee83-fea3-427d-aa75-a6ca4a2587ef admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:21.691966 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 974/1965] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:46:21 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:46:21.697646 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4413fe-a9c6-4c2e-9f0c-c959c9de7f31 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:21.710882 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4413fe-a9c6-4c2e-9f0c-c959c9de7f31 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:21.716661 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cbbfa28-9b92-4c98-9f8f-55a183ff887d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:21.717364 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbbfa28-9b92-4c98-9f8f-55a183ff887d admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:21.717642 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbbfa28-9b92-4c98-9f8f-55a183ff887d admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:21.717920 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbbfa28-9b92-4c98-9f8f-55a183ff887d admin admin] PATH_INFO: `/v3/projects/fa6c2b0055094b1b9f727f5fec00d6ce/users/0d36e579675c43549879eef4a59df651/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:21.779608 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cbbfa28-9b92-4c98-9f8f-55a183ff887d admin admin] RBAC: Authorizing `identity:create_grant(project_id=fa6c2b0055094b1b9f727f5fec00d6ce, user_id=0d36e579675c43549879eef4a59df651, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:21.785153 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cbbfa28-9b92-4c98-9f8f-55a183ff887d admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:21.844412 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ffee83-fea3-427d-aa75-a6ca4a2587ef admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:21.845423 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 992/1966] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:46:21 2022] PUT /identity/v3/projects/fa6c2b0055094b1b9f727f5fec00d6ce/users/0d36e579675c43549879eef4a59df651/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:21.857624 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ffee83-fea3-427d-aa75-a6ca4a2587ef admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:22.031143 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3db56f0-1855-4ce8-847d-30ac3c480a05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:22.032126 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3db56f0-1855-4ce8-847d-30ac3c480a05 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:22.033413 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3db56f0-1855-4ce8-847d-30ac3c480a05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:22.034959 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3db56f0-1855-4ce8-847d-30ac3c480a05 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:22.035832 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3db56f0-1855-4ce8-847d-30ac3c480a05 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:22.038324 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3db56f0-1855-4ce8-847d-30ac3c480a05 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:22.072879 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cbbfa28-9b92-4c98-9f8f-55a183ff887d admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:22.078381 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 975/1967] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:46:21 2022] GET /identity/v3/roles => generated 1420 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:46:22.087592 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cbbfa28-9b92-4c98-9f8f-55a183ff887d admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:22.105034 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35626762-904c-4d54-ae67-a6e33e1ceb61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:22.105736 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35626762-904c-4d54-ae67-a6e33e1ceb61 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:22.106017 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35626762-904c-4d54-ae67-a6e33e1ceb61 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:22.106292 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35626762-904c-4d54-ae67-a6e33e1ceb61 admin admin] PATH_INFO: `/v3/projects/fa6c2b0055094b1b9f727f5fec00d6ce/users/0d36e579675c43549879eef4a59df651/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:22.130730 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35626762-904c-4d54-ae67-a6e33e1ceb61 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fa6c2b0055094b1b9f727f5fec00d6ce, user_id=0d36e579675c43549879eef4a59df651, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:22.133481 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35626762-904c-4d54-ae67-a6e33e1ceb61 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:22.147606 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 993/1968] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:46:22 2022] PUT /identity/v3/projects/fa6c2b0055094b1b9f727f5fec00d6ce/users/0d36e579675c43549879eef4a59df651/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 77 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:22.157793 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5493562-e292-41eb-885e-4fe8788d7ca7 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:22.158094 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5493562-e292-41eb-885e-4fe8788d7ca7 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:22.158422 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5493562-e292-41eb-885e-4fe8788d7ca7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:22.194300 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-e5493562-e292-41eb-885e-4fe8788d7ca7 None None] MFA Rules not processed for user `0d36e579675c43549879eef4a59df651`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:22.195677 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-e5493562-e292-41eb-885e-4fe8788d7ca7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:22.319996 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 976/1969] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:22 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:46:22.331920 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35626762-904c-4d54-ae67-a6e33e1ceb61 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:22.338909 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-35626762-904c-4d54-ae67-a6e33e1ceb61 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:22.399117 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:22.434105 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35626762-904c-4d54-ae67-a6e33e1ceb61 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:22.440644 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa93758-3585-48dc-ba8c-54d9d9ffbd3e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:22.441574 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa93758-3585-48dc-ba8c-54d9d9ffbd3e service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:22.443021 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa93758-3585-48dc-ba8c-54d9d9ffbd3e service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:22.443373 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa93758-3585-48dc-ba8c-54d9d9ffbd3e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:22.459888 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa93758-3585-48dc-ba8c-54d9d9ffbd3e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:22.463041 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa93758-3585-48dc-ba8c-54d9d9ffbd3e service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:22.480680 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 994/1970] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:46:22 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:46:22.656497 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5493562-e292-41eb-885e-4fe8788d7ca7 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:22.674122 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5493562-e292-41eb-885e-4fe8788d7ca7 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:22.679800 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8368006e-3a33-4a93-8d6c-beeff91c4524 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:22.680363 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8368006e-3a33-4a93-8d6c-beeff91c4524 service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:22.680645 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8368006e-3a33-4a93-8d6c-beeff91c4524 service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:22.680920 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8368006e-3a33-4a93-8d6c-beeff91c4524 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:22.755240 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:22.822262 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8368006e-3a33-4a93-8d6c-beeff91c4524 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:22.824150 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8368006e-3a33-4a93-8d6c-beeff91c4524 service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:22.836449 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 977/1971] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:46:22 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:46:23.221725 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa93758-3585-48dc-ba8c-54d9d9ffbd3e service cinder] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:23.352272 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa93758-3585-48dc-ba8c-54d9d9ffbd3e service cinder] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:23.363225 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca082a6-178e-47a4-8c61-2e3dbd32d337 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:23.364868 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca082a6-178e-47a4-8c61-2e3dbd32d337 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:23.365354 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca082a6-178e-47a4-8c61-2e3dbd32d337 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:23.366008 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca082a6-178e-47a4-8c61-2e3dbd32d337 admin admin] PATH_INFO: `/v3/users/da9f6060b0e848aea19e89ebe3c9bed6` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:23.369795 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca082a6-178e-47a4-8c61-2e3dbd32d337 admin admin] RBAC: Authorizing `identity:delete_user(user_id=da9f6060b0e848aea19e89ebe3c9bed6)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:23.372691 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca082a6-178e-47a4-8c61-2e3dbd32d337 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:23.464358 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7ca082a6-178e-47a4-8c61-2e3dbd32d337 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'da9f6060b0e848aea19e89ebe3c9bed6'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:23.475198 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7ca082a6-178e-47a4-8c61-2e3dbd32d337 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'da9f6060b0e848aea19e89ebe3c9bed6'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:23.475856 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7ca082a6-178e-47a4-8c61-2e3dbd32d337 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'da9f6060b0e848aea19e89ebe3c9bed6'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:23.476523 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7ca082a6-178e-47a4-8c61-2e3dbd32d337 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'da9f6060b0e848aea19e89ebe3c9bed6'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:23.488673 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7ca082a6-178e-47a4-8c61-2e3dbd32d337 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'da9f6060b0e848aea19e89ebe3c9bed6'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:23.514222 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 995/1972] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:46:23 2022] DELETE /identity/v3/users/da9f6060b0e848aea19e89ebe3c9bed6 => generated 0 bytes in 293 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:23.518962 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171446fb-b0ff-4138-84a6-1e8df1cbc6ca None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:23.519817 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171446fb-b0ff-4138-84a6-1e8df1cbc6ca None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:23.519981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171446fb-b0ff-4138-84a6-1e8df1cbc6ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:23.564243 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-171446fb-b0ff-4138-84a6-1e8df1cbc6ca None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:23.567082 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-171446fb-b0ff-4138-84a6-1e8df1cbc6ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:23.744310 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 978/1973] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 226 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:46:23.754089 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca082a6-178e-47a4-8c61-2e3dbd32d337 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:23.756485 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7ca082a6-178e-47a4-8c61-2e3dbd32d337 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:23.924713 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca082a6-178e-47a4-8c61-2e3dbd32d337 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:23.936461 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e172d96c-006a-465a-98e0-31feeef5d09b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:23.937109 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e172d96c-006a-465a-98e0-31feeef5d09b service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:23.937388 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e172d96c-006a-465a-98e0-31feeef5d09b service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:23.937663 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e172d96c-006a-465a-98e0-31feeef5d09b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:23.962650 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e172d96c-006a-465a-98e0-31feeef5d09b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:23.962650 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e172d96c-006a-465a-98e0-31feeef5d09b service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:23.962650 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:23.981655 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 996/1974] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:46:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 224 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:46:24.368706 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-171446fb-b0ff-4138-84a6-1e8df1cbc6ca None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:24.371723 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-171446fb-b0ff-4138-84a6-1e8df1cbc6ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:24.396808 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-171446fb-b0ff-4138-84a6-1e8df1cbc6ca None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:24.412681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd829da7-cfdf-497e-9fd9-b77757884255 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:24.412961 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd829da7-cfdf-497e-9fd9-b77757884255 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:24.413220 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd829da7-cfdf-497e-9fd9-b77757884255 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:24.413386 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd829da7-cfdf-497e-9fd9-b77757884255 admin admin] PATH_INFO: `/v3/users/21fcf2b0fd3b43cdac8a682a8e8b2f87` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:24.415941 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd829da7-cfdf-497e-9fd9-b77757884255 admin admin] RBAC: Authorizing `identity:delete_user(user_id=21fcf2b0fd3b43cdac8a682a8e8b2f87)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:24.418109 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd829da7-cfdf-497e-9fd9-b77757884255 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:24.537123 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-cd829da7-cfdf-497e-9fd9-b77757884255 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '21fcf2b0fd3b43cdac8a682a8e8b2f87'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:24.553917 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-cd829da7-cfdf-497e-9fd9-b77757884255 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '21fcf2b0fd3b43cdac8a682a8e8b2f87'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:24.558833 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-cd829da7-cfdf-497e-9fd9-b77757884255 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '21fcf2b0fd3b43cdac8a682a8e8b2f87'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:24.560884 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-cd829da7-cfdf-497e-9fd9-b77757884255 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '21fcf2b0fd3b43cdac8a682a8e8b2f87'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:24.581159 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-cd829da7-cfdf-497e-9fd9-b77757884255 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '21fcf2b0fd3b43cdac8a682a8e8b2f87'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:24.617407 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 979/1975] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:46:24 2022] DELETE /identity/v3/users/21fcf2b0fd3b43cdac8a682a8e8b2f87 => generated 0 bytes in 250 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:24.623146 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e353f6-101c-4504-b1a7-b741a7f9fdfe None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:24.623732 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e353f6-101c-4504-b1a7-b741a7f9fdfe None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:24.623732 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e353f6-101c-4504-b1a7-b741a7f9fdfe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:24.652015 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd829da7-cfdf-497e-9fd9-b77757884255 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:24.653621 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-cd829da7-cfdf-497e-9fd9-b77757884255 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:24.667286 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-55e353f6-101c-4504-b1a7-b741a7f9fdfe None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:24.668524 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-55e353f6-101c-4504-b1a7-b741a7f9fdfe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:24.763774 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:24.821877 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 997/1976] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:46:24.838963 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd829da7-cfdf-497e-9fd9-b77757884255 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:24.852059 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d66039-2458-4c13-bad1-05ae0b7cdd58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:24.852988 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d66039-2458-4c13-bad1-05ae0b7cdd58 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:24.853414 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d66039-2458-4c13-bad1-05ae0b7cdd58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:24.853865 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d66039-2458-4c13-bad1-05ae0b7cdd58 admin admin] PATH_INFO: `/v3/projects/00e9f9d45a7c46e4a65aa6afa1e49778` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:24.855491 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d66039-2458-4c13-bad1-05ae0b7cdd58 admin admin] RBAC: Authorizing `identity:delete_project(project_id=00e9f9d45a7c46e4a65aa6afa1e49778)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:24.858658 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d66039-2458-4c13-bad1-05ae0b7cdd58 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:24.893300 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e353f6-101c-4504-b1a7-b741a7f9fdfe None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:24.894744 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-55e353f6-101c-4504-b1a7-b741a7f9fdfe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:24.976845 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-17d66039-2458-4c13-bad1-05ae0b7cdd58 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '00e9f9d45a7c46e4a65aa6afa1e49778', 'request_id': 'req-17d66039-2458-4c13-bad1-05ae0b7cdd58'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:24.985117 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e353f6-101c-4504-b1a7-b741a7f9fdfe None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:24.990031 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-17d66039-2458-4c13-bad1-05ae0b7cdd58 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '00e9f9d45a7c46e4a65aa6afa1e49778', 'request_id': 'req-17d66039-2458-4c13-bad1-05ae0b7cdd58'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:25.008049 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-17d66039-2458-4c13-bad1-05ae0b7cdd58 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '00e9f9d45a7c46e4a65aa6afa1e49778', 'request_id': 'req-17d66039-2458-4c13-bad1-05ae0b7cdd58'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:25.035372 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-17d66039-2458-4c13-bad1-05ae0b7cdd58 admin admin] The token cache is being invalidate because project 00e9f9d45a7c46e4a65aa6afa1e49778 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:46:25.035372 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-17d66039-2458-4c13-bad1-05ae0b7cdd58 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:25.039229 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-17d66039-2458-4c13-bad1-05ae0b7cdd58 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:25.044632 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 980/1977] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:46:24 2022] DELETE /identity/v3/projects/00e9f9d45a7c46e4a65aa6afa1e49778 => generated 0 bytes in 390 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:25.047953 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d66039-2458-4c13-bad1-05ae0b7cdd58 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:25.051631 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-17d66039-2458-4c13-bad1-05ae0b7cdd58 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:25.078872 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9df2d9a-c036-471c-be8f-c0fef952203a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:25.079760 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9df2d9a-c036-471c-be8f-c0fef952203a service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:25.080125 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9df2d9a-c036-471c-be8f-c0fef952203a service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:25.080485 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9df2d9a-c036-471c-be8f-c0fef952203a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:25.082739 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-c9df2d9a-c036-471c-be8f-c0fef952203a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:25.193635 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d66039-2458-4c13-bad1-05ae0b7cdd58 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:25.207278 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76a4ad7-ad9d-4093-8bb9-64eb19e30be0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:25.208555 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76a4ad7-ad9d-4093-8bb9-64eb19e30be0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:25.208848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76a4ad7-ad9d-4093-8bb9-64eb19e30be0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:25.209124 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76a4ad7-ad9d-4093-8bb9-64eb19e30be0 admin admin] PATH_INFO: `/v3/users/0d36e579675c43549879eef4a59df651` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:25.212027 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a76a4ad7-ad9d-4093-8bb9-64eb19e30be0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0d36e579675c43549879eef4a59df651)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:25.215578 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a76a4ad7-ad9d-4093-8bb9-64eb19e30be0 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:25.217137 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9df2d9a-c036-471c-be8f-c0fef952203a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:25.218771 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9df2d9a-c036-471c-be8f-c0fef952203a service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:25.232058 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 998/1978] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:46:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:46:25.323484 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-a76a4ad7-ad9d-4093-8bb9-64eb19e30be0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0d36e579675c43549879eef4a59df651'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:25.339088 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-a76a4ad7-ad9d-4093-8bb9-64eb19e30be0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0d36e579675c43549879eef4a59df651'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:25.340578 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-a76a4ad7-ad9d-4093-8bb9-64eb19e30be0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0d36e579675c43549879eef4a59df651'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:25.342830 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-a76a4ad7-ad9d-4093-8bb9-64eb19e30be0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0d36e579675c43549879eef4a59df651'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:25.359082 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-a76a4ad7-ad9d-4093-8bb9-64eb19e30be0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0d36e579675c43549879eef4a59df651'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:25.366971 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:25.388530 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 981/1979] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:46:25 2022] DELETE /identity/v3/users/0d36e579675c43549879eef4a59df651 => generated 0 bytes in 343 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:25.810321 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9df2d9a-c036-471c-be8f-c0fef952203a service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:25.821251 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-c9df2d9a-c036-471c-be8f-c0fef952203a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:25.846899 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9df2d9a-c036-471c-be8f-c0fef952203a service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:25.883565 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:26.097404 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cfe9281-b000-4d0c-986d-6271e6201b00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:26.098087 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfe9281-b000-4d0c-986d-6271e6201b00 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:26.098373 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfe9281-b000-4d0c-986d-6271e6201b00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:26.108830 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfe9281-b000-4d0c-986d-6271e6201b00 admin admin] PATH_INFO: `/v3/projects/fd9693ee1e254baa9bd612828fdeab20` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:26.113765 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cfe9281-b000-4d0c-986d-6271e6201b00 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fd9693ee1e254baa9bd612828fdeab20)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:26.116938 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cfe9281-b000-4d0c-986d-6271e6201b00 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:26.216162 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-9cfe9281-b000-4d0c-986d-6271e6201b00 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fd9693ee1e254baa9bd612828fdeab20', 'request_id': 'req-9cfe9281-b000-4d0c-986d-6271e6201b00'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:26.225941 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-9cfe9281-b000-4d0c-986d-6271e6201b00 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fd9693ee1e254baa9bd612828fdeab20', 'request_id': 'req-9cfe9281-b000-4d0c-986d-6271e6201b00'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:26.242996 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-9cfe9281-b000-4d0c-986d-6271e6201b00 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fd9693ee1e254baa9bd612828fdeab20', 'request_id': 'req-9cfe9281-b000-4d0c-986d-6271e6201b00'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:26.259160 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-9cfe9281-b000-4d0c-986d-6271e6201b00 admin admin] The token cache is being invalidate because project fd9693ee1e254baa9bd612828fdeab20 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:46:26.259770 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-9cfe9281-b000-4d0c-986d-6271e6201b00 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:26.281513 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-9cfe9281-b000-4d0c-986d-6271e6201b00 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:26.291845 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 999/1980] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:46:25 2022] DELETE /identity/v3/projects/fd9693ee1e254baa9bd612828fdeab20 => generated 0 bytes in 482 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:26.719338 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76a4ad7-ad9d-4093-8bb9-64eb19e30be0 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:26.721634 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-a76a4ad7-ad9d-4093-8bb9-64eb19e30be0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:26.736664 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76a4ad7-ad9d-4093-8bb9-64eb19e30be0 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:26.827200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d81af22-917b-4ed5-9d53-81cc954c9f19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:26.827950 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d81af22-917b-4ed5-9d53-81cc954c9f19 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:26.828335 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d81af22-917b-4ed5-9d53-81cc954c9f19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:26.828714 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d81af22-917b-4ed5-9d53-81cc954c9f19 admin admin] PATH_INFO: `/v3/projects/fa6c2b0055094b1b9f727f5fec00d6ce` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:26.830126 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d81af22-917b-4ed5-9d53-81cc954c9f19 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fa6c2b0055094b1b9f727f5fec00d6ce)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:26.832157 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d81af22-917b-4ed5-9d53-81cc954c9f19 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:26.880152 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b946879-30ec-4885-87b9-d2b320f04f84 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:26.880711 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b946879-30ec-4885-87b9-d2b320f04f84 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:26.880711 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b946879-30ec-4885-87b9-d2b320f04f84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:26.896141 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-3d81af22-917b-4ed5-9d53-81cc954c9f19 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fa6c2b0055094b1b9f727f5fec00d6ce', 'request_id': 'req-3d81af22-917b-4ed5-9d53-81cc954c9f19'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:26.904530 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-3d81af22-917b-4ed5-9d53-81cc954c9f19 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fa6c2b0055094b1b9f727f5fec00d6ce', 'request_id': 'req-3d81af22-917b-4ed5-9d53-81cc954c9f19'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:26.919167 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-6b946879-30ec-4885-87b9-d2b320f04f84 None None] MFA Rules not processed for user `eb94858904f14e0aa1d31a2a5f6c48b1`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:26.921055 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-3d81af22-917b-4ed5-9d53-81cc954c9f19 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fa6c2b0055094b1b9f727f5fec00d6ce', 'request_id': 'req-3d81af22-917b-4ed5-9d53-81cc954c9f19'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:26.921390 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-6b946879-30ec-4885-87b9-d2b320f04f84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:26.935640 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-3d81af22-917b-4ed5-9d53-81cc954c9f19 admin admin] The token cache is being invalidate because project fa6c2b0055094b1b9f727f5fec00d6ce was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:46:26.936086 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-3d81af22-917b-4ed5-9d53-81cc954c9f19 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:26.937346 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-3d81af22-917b-4ed5-9d53-81cc954c9f19 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:26.943252 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 982/1981] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:46:26 2022] DELETE /identity/v3/projects/fa6c2b0055094b1b9f727f5fec00d6ce => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:26.944974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f06063-1b2f-408f-8314-5cba15bc5192 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:26.944974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f06063-1b2f-408f-8314-5cba15bc5192 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:26.945339 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f06063-1b2f-408f-8314-5cba15bc5192 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:26.969297 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-01f06063-1b2f-408f-8314-5cba15bc5192 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:26.970761 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-01f06063-1b2f-408f-8314-5cba15bc5192 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:27.025757 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1000/1982] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:26 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:46:27.038440 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b946879-30ec-4885-87b9-d2b320f04f84 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:27.039973 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-6b946879-30ec-4885-87b9-d2b320f04f84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:27.043848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 983/1983] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:46:27.052131 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcc129e-5db0-40c0-aea3-4139d3ddf5df None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:27.052550 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcc129e-5db0-40c0-aea3-4139d3ddf5df None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:27.052874 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcc129e-5db0-40c0-aea3-4139d3ddf5df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:27.070767 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:27.078502 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-bfcc129e-5db0-40c0-aea3-4139d3ddf5df None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:27.080276 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-bfcc129e-5db0-40c0-aea3-4139d3ddf5df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:27.094289 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 984/1984] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:46:27.095212 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfcc129e-5db0-40c0-aea3-4139d3ddf5df None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:27.104442 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfcc129e-5db0-40c0-aea3-4139d3ddf5df None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:27.110829 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64365fed-ebc1-4116-855a-ab9eee6b52d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:27.111449 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64365fed-ebc1-4116-855a-ab9eee6b52d4 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:27.111727 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64365fed-ebc1-4116-855a-ab9eee6b52d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:27.111998 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64365fed-ebc1-4116-855a-ab9eee6b52d4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:27.112645 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64365fed-ebc1-4116-855a-ab9eee6b52d4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:46:27.112951 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64365fed-ebc1-4116-855a-ab9eee6b52d4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:27.115226 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64365fed-ebc1-4116-855a-ab9eee6b52d4 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:27.117857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:DELETE=9,keystone:SELECT=90,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:27.129158 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 985/1985] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:46:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:46:27.134102 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64365fed-ebc1-4116-855a-ab9eee6b52d4 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:27.135800 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b946879-30ec-4885-87b9-d2b320f04f84 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:27.139665 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47656984-6bdc-4853-be3c-a393315bc7c6 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4b2ea9aa31445d69e0c1d96b5657708', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:27.140279 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47656984-6bdc-4853-be3c-a393315bc7c6 service glance] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:27.140562 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47656984-6bdc-4853-be3c-a393315bc7c6 service glance] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:27.140843 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47656984-6bdc-4853-be3c-a393315bc7c6 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:27.142640 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-47656984-6bdc-4853-be3c-a393315bc7c6 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:27.145186 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64365fed-ebc1-4116-855a-ab9eee6b52d4 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:27.150655 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd13a89-7f72-487b-8077-9dc359cc8df3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:27.151399 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd13a89-7f72-487b-8077-9dc359cc8df3 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:27.151772 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd13a89-7f72-487b-8077-9dc359cc8df3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:27.152146 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd13a89-7f72-487b-8077-9dc359cc8df3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:27.152884 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd13a89-7f72-487b-8077-9dc359cc8df3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:27.154914 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd13a89-7f72-487b-8077-9dc359cc8df3 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:27.190565 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 986/1986] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:46:27 2022] POST /identity/v3/projects => generated 371 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:46:27.193219 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd13a89-7f72-487b-8077-9dc359cc8df3 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:27.216970 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47656984-6bdc-4853-be3c-a393315bc7c6 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:27.218504 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47656984-6bdc-4853-be3c-a393315bc7c6 service glance] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:27.238730 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1001/1987] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:46:27 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 201 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:46:27.260537 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd13a89-7f72-487b-8077-9dc359cc8df3 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:27.265269 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b0d7a2-c997-41ac-a34a-06a906190005 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:27.265896 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b0d7a2-c997-41ac-a34a-06a906190005 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:27.266178 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b0d7a2-c997-41ac-a34a-06a906190005 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:27.266483 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b0d7a2-c997-41ac-a34a-06a906190005 admin admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:27.267135 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b0d7a2-c997-41ac-a34a-06a906190005 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:27.269068 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b0d7a2-c997-41ac-a34a-06a906190005 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:27.294485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 987/1988] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:46:27 2022] POST /identity/v3/users => generated 286 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:46:27.297924 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47656984-6bdc-4853-be3c-a393315bc7c6 service glance] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:27.311407 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47656984-6bdc-4853-be3c-a393315bc7c6 service glance] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:27.316662 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3d31d1-6db7-46de-ac22-c2dbac206e8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:27.317900 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3d31d1-6db7-46de-ac22-c2dbac206e8c admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:27.318192 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3d31d1-6db7-46de-ac22-c2dbac206e8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:27.324271 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f3d31d1-6db7-46de-ac22-c2dbac206e8c admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:27.324271 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f3d31d1-6db7-46de-ac22-c2dbac206e8c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:27.324271 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f3d31d1-6db7-46de-ac22-c2dbac206e8c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:27.338457 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b0d7a2-c997-41ac-a34a-06a906190005 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:27.338681 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1002/1989] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:46:27 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:46:27.350618 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b0d7a2-c997-41ac-a34a-06a906190005 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:27.355833 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c74bdc6-0dbc-4762-8662-0c4d527a462c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:27.356497 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c74bdc6-0dbc-4762-8662-0c4d527a462c admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:27.356779 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c74bdc6-0dbc-4762-8662-0c4d527a462c admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:27.357116 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c74bdc6-0dbc-4762-8662-0c4d527a462c admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:27.357707 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c74bdc6-0dbc-4762-8662-0c4d527a462c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:27.359433 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c74bdc6-0dbc-4762-8662-0c4d527a462c admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:27.370363 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 988/1990] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:46:27 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:46:27.374634 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3d31d1-6db7-46de-ac22-c2dbac206e8c admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:27.386527 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f3d31d1-6db7-46de-ac22-c2dbac206e8c admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:27.387813 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d41d7dc-6f99-47e3-9ccd-bc5aab1114af None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:27.388102 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d41d7dc-6f99-47e3-9ccd-bc5aab1114af None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:27.389219 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d41d7dc-6f99-47e3-9ccd-bc5aab1114af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:27.391909 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d5cb02-bca6-49e4-95b6-7a4d34746448 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:27.392601 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d5cb02-bca6-49e4-95b6-7a4d34746448 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:27.392841 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d5cb02-bca6-49e4-95b6-7a4d34746448 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:27.393118 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d5cb02-bca6-49e4-95b6-7a4d34746448 admin admin] PATH_INFO: `/v3/projects/118fbfbd6f844591a149078bdb60ae16/users/e13212a106cf446d9bf2777c7030ca76/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:27.416529 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-1d41d7dc-6f99-47e3-9ccd-bc5aab1114af None None] MFA Rules not processed for user `97c8877c74194a6bbbf25b3c8f5b7dbd`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:27.417817 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79d5cb02-bca6-49e4-95b6-7a4d34746448 admin admin] RBAC: Authorizing `identity:create_grant(project_id=118fbfbd6f844591a149078bdb60ae16, user_id=e13212a106cf446d9bf2777c7030ca76, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:27.419240 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-1d41d7dc-6f99-47e3-9ccd-bc5aab1114af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:27.421396 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79d5cb02-bca6-49e4-95b6-7a4d34746448 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:27.441703 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1003/1991] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:46:27 2022] PUT /identity/v3/projects/118fbfbd6f844591a149078bdb60ae16/users/e13212a106cf446d9bf2777c7030ca76/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:27.454187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20793323-efba-4258-8048-b993c614f8c7 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:27.454784 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20793323-efba-4258-8048-b993c614f8c7 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:27.457498 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20793323-efba-4258-8048-b993c614f8c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:27.509162 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-20793323-efba-4258-8048-b993c614f8c7 None None] MFA Rules not processed for user `e13212a106cf446d9bf2777c7030ca76`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:27.510822 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-20793323-efba-4258-8048-b993c614f8c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:27.511998 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 989/1992] 10.222.0.25 () {58 vars in 1009 bytes} [Fri Oct 28 09:46:27 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:46:27.568888 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d41d7dc-6f99-47e3-9ccd-bc5aab1114af None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:27.576672 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-1d41d7dc-6f99-47e3-9ccd-bc5aab1114af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:27.610420 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1004/1993] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:27 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:46:27.620599 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20793323-efba-4258-8048-b993c614f8c7 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:27.622553 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-20793323-efba-4258-8048-b993c614f8c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:27.642243 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:27.651154 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:27.671782 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d41d7dc-6f99-47e3-9ccd-bc5aab1114af None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:27.705330 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20793323-efba-4258-8048-b993c614f8c7 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:27.711273 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55539154-7c28-4cdf-8692-25617119fc2f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:27.711908 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55539154-7c28-4cdf-8692-25617119fc2f service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:27.712188 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55539154-7c28-4cdf-8692-25617119fc2f service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:27.712462 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55539154-7c28-4cdf-8692-25617119fc2f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:27.723716 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55539154-7c28-4cdf-8692-25617119fc2f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:27.740103 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55539154-7c28-4cdf-8692-25617119fc2f service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:27.751481 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1005/1994] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:46:27 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:46:27.787691 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55539154-7c28-4cdf-8692-25617119fc2f service cinder] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:27.859408 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f9af8e-a4ba-45c7-9fad-8d6fb46a0b79 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c6cffe0d56e45c4b8fb54fbe5df6094', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:27.860200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f9af8e-a4ba-45c7-9fad-8d6fb46a0b79 service swift] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:27.860584 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f9af8e-a4ba-45c7-9fad-8d6fb46a0b79 service swift] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:27.860962 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f9af8e-a4ba-45c7-9fad-8d6fb46a0b79 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:27.919688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55539154-7c28-4cdf-8692-25617119fc2f service cinder] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:27.928839 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a48f375-62ff-41c6-b0ea-19e54d1347a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:27.929506 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a48f375-62ff-41c6-b0ea-19e54d1347a3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:27.929990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a48f375-62ff-41c6-b0ea-19e54d1347a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:27.930270 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a48f375-62ff-41c6-b0ea-19e54d1347a3 admin admin] PATH_INFO: `/v3/users/e13212a106cf446d9bf2777c7030ca76` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:27.934582 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a48f375-62ff-41c6-b0ea-19e54d1347a3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e13212a106cf446d9bf2777c7030ca76)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:27.936533 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a48f375-62ff-41c6-b0ea-19e54d1347a3 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:27.986192 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f9af8e-a4ba-45c7-9fad-8d6fb46a0b79 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:27.988192 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f9af8e-a4ba-45c7-9fad-8d6fb46a0b79 service swift] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:28.009332 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 990/1995] 10.222.0.25 () {60 vars in 1384 bytes} [Fri Oct 28 09:46:27 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 441 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 28 09:46:28.079257 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-4a48f375-62ff-41c6-b0ea-19e54d1347a3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e13212a106cf446d9bf2777c7030ca76'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:28.100119 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-4a48f375-62ff-41c6-b0ea-19e54d1347a3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e13212a106cf446d9bf2777c7030ca76'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:28.101302 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-4a48f375-62ff-41c6-b0ea-19e54d1347a3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e13212a106cf446d9bf2777c7030ca76'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:28.102994 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-4a48f375-62ff-41c6-b0ea-19e54d1347a3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e13212a106cf446d9bf2777c7030ca76'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:28.126916 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-4a48f375-62ff-41c6-b0ea-19e54d1347a3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e13212a106cf446d9bf2777c7030ca76'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:28.178553 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1006/1996] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:46:27 2022] DELETE /identity/v3/users/e13212a106cf446d9bf2777c7030ca76 => generated 0 bytes in 399 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:28.182973 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21cc931-2d16-4638-ba89-33e77da86872 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:28.183330 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21cc931-2d16-4638-ba89-33e77da86872 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:28.183668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21cc931-2d16-4638-ba89-33e77da86872 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:28.213947 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-c21cc931-2d16-4638-ba89-33e77da86872 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:28.215560 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-c21cc931-2d16-4638-ba89-33e77da86872 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:28.360488 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 991/1997] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:46:28.370564 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a48f375-62ff-41c6-b0ea-19e54d1347a3 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:28.372780 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4a48f375-62ff-41c6-b0ea-19e54d1347a3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:28.427663 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:28.506047 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a48f375-62ff-41c6-b0ea-19e54d1347a3 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:28.515263 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c124b82c-308e-4e3a-ba6d-61a1ba8b501a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:28.515908 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c124b82c-308e-4e3a-ba6d-61a1ba8b501a service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:28.516202 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c124b82c-308e-4e3a-ba6d-61a1ba8b501a service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:28.516495 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c124b82c-308e-4e3a-ba6d-61a1ba8b501a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:28.544515 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c124b82c-308e-4e3a-ba6d-61a1ba8b501a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:28.547506 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c124b82c-308e-4e3a-ba6d-61a1ba8b501a service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:28.579727 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1007/1998] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:46:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 210 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:46:29.385350 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c21cc931-2d16-4638-ba89-33e77da86872 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:29.387481 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-c21cc931-2d16-4638-ba89-33e77da86872 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:29.408199 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c21cc931-2d16-4638-ba89-33e77da86872 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:29.423928 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-389de9be-14bd-43d6-a903-04ab68a6e6fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:29.424604 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389de9be-14bd-43d6-a903-04ab68a6e6fb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:29.424888 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389de9be-14bd-43d6-a903-04ab68a6e6fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:29.425160 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389de9be-14bd-43d6-a903-04ab68a6e6fb admin admin] PATH_INFO: `/v3/projects/118fbfbd6f844591a149078bdb60ae16` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:29.428983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-389de9be-14bd-43d6-a903-04ab68a6e6fb admin admin] RBAC: Authorizing `identity:delete_project(project_id=118fbfbd6f844591a149078bdb60ae16)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:29.431399 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-389de9be-14bd-43d6-a903-04ab68a6e6fb admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:29.516290 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:29.532112 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-389de9be-14bd-43d6-a903-04ab68a6e6fb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '118fbfbd6f844591a149078bdb60ae16', 'request_id': 'req-389de9be-14bd-43d6-a903-04ab68a6e6fb'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:29.541355 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-389de9be-14bd-43d6-a903-04ab68a6e6fb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '118fbfbd6f844591a149078bdb60ae16', 'request_id': 'req-389de9be-14bd-43d6-a903-04ab68a6e6fb'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:29.553532 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-389de9be-14bd-43d6-a903-04ab68a6e6fb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '118fbfbd6f844591a149078bdb60ae16', 'request_id': 'req-389de9be-14bd-43d6-a903-04ab68a6e6fb'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:29.570799 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-389de9be-14bd-43d6-a903-04ab68a6e6fb admin admin] The token cache is being invalidate because project 118fbfbd6f844591a149078bdb60ae16 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:46:29.571156 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-389de9be-14bd-43d6-a903-04ab68a6e6fb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:29.571822 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-389de9be-14bd-43d6-a903-04ab68a6e6fb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:29.577565 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 992/1999] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:46:29 2022] DELETE /identity/v3/projects/118fbfbd6f844591a149078bdb60ae16 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:29.578282 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460a6426-7613-4492-9a34-56f644b0e7c8 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:29.578771 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460a6426-7613-4492-9a34-56f644b0e7c8 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:29.579057 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460a6426-7613-4492-9a34-56f644b0e7c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:29.609085 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-460a6426-7613-4492-9a34-56f644b0e7c8 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:29.612092 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-460a6426-7613-4492-9a34-56f644b0e7c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:29.724545 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1008/2000] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:46:29.730717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01186a4-663c-4171-bbda-391daa839494 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:29.731221 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01186a4-663c-4171-bbda-391daa839494 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:29.732154 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b01186a4-663c-4171-bbda-391daa839494 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:29.780304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-b01186a4-663c-4171-bbda-391daa839494 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:29.783247 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-b01186a4-663c-4171-bbda-391daa839494 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:29.810478 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 993/2001] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:46:29.814271 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460a6426-7613-4492-9a34-56f644b0e7c8 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:29.849795 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460a6426-7613-4492-9a34-56f644b0e7c8 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:29.870533 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddcd9a12-ca42-494b-a7d1-4b7efab1f9cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:29.871603 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddcd9a12-ca42-494b-a7d1-4b7efab1f9cb admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:29.872441 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddcd9a12-ca42-494b-a7d1-4b7efab1f9cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:29.873258 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddcd9a12-ca42-494b-a7d1-4b7efab1f9cb admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:29.874511 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddcd9a12-ca42-494b-a7d1-4b7efab1f9cb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:46:29.875145 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddcd9a12-ca42-494b-a7d1-4b7efab1f9cb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:29.886132 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddcd9a12-ca42-494b-a7d1-4b7efab1f9cb admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:29.944915 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1009/2002] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:46:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 132 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:46:29.949206 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01186a4-663c-4171-bbda-391daa839494 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:29.970727 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b01186a4-663c-4171-bbda-391daa839494 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:29.990600 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf59f55-69a4-4c28-aa4d-109c7f763c39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:29.991045 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf59f55-69a4-4c28-aa4d-109c7f763c39 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:29.994477 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf59f55-69a4-4c28-aa4d-109c7f763c39 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:29.994477 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf59f55-69a4-4c28-aa4d-109c7f763c39 admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:29.994477 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdf59f55-69a4-4c28-aa4d-109c7f763c39 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:30.021475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdf59f55-69a4-4c28-aa4d-109c7f763c39 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:30.114658 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 994/2003] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:46:29 2022] POST /identity/v3/projects => generated 367 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:46:30.118273 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddcd9a12-ca42-494b-a7d1-4b7efab1f9cb admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:30.149271 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddcd9a12-ca42-494b-a7d1-4b7efab1f9cb admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:30.390482 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e95044e-888e-486b-81b7-998fda92ae08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:30.391203 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e95044e-888e-486b-81b7-998fda92ae08 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:30.391583 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e95044e-888e-486b-81b7-998fda92ae08 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:30.391960 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e95044e-888e-486b-81b7-998fda92ae08 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:30.392706 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e95044e-888e-486b-81b7-998fda92ae08 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:30.399193 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e95044e-888e-486b-81b7-998fda92ae08 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:30.453129 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1010/2004] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:46:30 2022] POST /identity/v3/users => generated 284 bytes in 336 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:46:30.454888 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf59f55-69a4-4c28-aa4d-109c7f763c39 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:30.481938 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf59f55-69a4-4c28-aa4d-109c7f763c39 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:30.503619 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5e2af8-fcc2-47e0-bd43-8884464eb45a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:30.504332 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5e2af8-fcc2-47e0-bd43-8884464eb45a admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:30.505336 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5e2af8-fcc2-47e0-bd43-8884464eb45a admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:30.505738 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5e2af8-fcc2-47e0-bd43-8884464eb45a admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:30.506930 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f5e2af8-fcc2-47e0-bd43-8884464eb45a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:30.515099 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f5e2af8-fcc2-47e0-bd43-8884464eb45a admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:30.564046 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 995/2005] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:46:30 2022] GET /identity/v3/roles => generated 1420 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:46:30.568682 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e95044e-888e-486b-81b7-998fda92ae08 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:30.581817 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e95044e-888e-486b-81b7-998fda92ae08 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:30.588852 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dba7f704-79c1-4250-b20a-5837e455f834 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:30.589905 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba7f704-79c1-4250-b20a-5837e455f834 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:30.590729 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba7f704-79c1-4250-b20a-5837e455f834 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:30.591236 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba7f704-79c1-4250-b20a-5837e455f834 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:30.592118 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba7f704-79c1-4250-b20a-5837e455f834 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:30.594015 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dba7f704-79c1-4250-b20a-5837e455f834 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:30.627369 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1011/2006] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:46:30 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:46:30.628899 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5e2af8-fcc2-47e0-bd43-8884464eb45a admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:30.639712 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5e2af8-fcc2-47e0-bd43-8884464eb45a admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:30.649933 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb1bd608-9c1d-4fac-aab7-4dec25c827f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:30.651293 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1bd608-9c1d-4fac-aab7-4dec25c827f5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:30.651846 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1bd608-9c1d-4fac-aab7-4dec25c827f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:30.652780 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1bd608-9c1d-4fac-aab7-4dec25c827f5 admin admin] PATH_INFO: `/v3/projects/e755446b01eb49f081e9ce5b2442b503/users/802938c8b66c4ee59d8ba9c3bdc5c534/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:30.683659 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb1bd608-9c1d-4fac-aab7-4dec25c827f5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e755446b01eb49f081e9ce5b2442b503, user_id=802938c8b66c4ee59d8ba9c3bdc5c534, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:30.689856 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb1bd608-9c1d-4fac-aab7-4dec25c827f5 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:30.722920 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 996/2007] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:46:30 2022] PUT /identity/v3/projects/e755446b01eb49f081e9ce5b2442b503/users/802938c8b66c4ee59d8ba9c3bdc5c534/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 95 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:30.740065 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-865e4fa9-d76d-4512-aad8-97bef8533720 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:30.740697 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-865e4fa9-d76d-4512-aad8-97bef8533720 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:30.741185 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-865e4fa9-d76d-4512-aad8-97bef8533720 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:30.826538 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-865e4fa9-d76d-4512-aad8-97bef8533720 None None] MFA Rules not processed for user `802938c8b66c4ee59d8ba9c3bdc5c534`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:30.828138 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-865e4fa9-d76d-4512-aad8-97bef8533720 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:30.921764 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:30.995981 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1012/2008] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:30 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 258 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:46:31.020696 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb1bd608-9c1d-4fac-aab7-4dec25c827f5 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:31.023016 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-eb1bd608-9c1d-4fac-aab7-4dec25c827f5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:31.094937 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb1bd608-9c1d-4fac-aab7-4dec25c827f5 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:31.099389 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94534b5b-95fc-4041-abfe-ce03a8a318c2 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4b2ea9aa31445d69e0c1d96b5657708', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:31.100138 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94534b5b-95fc-4041-abfe-ce03a8a318c2 service glance] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:31.100521 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94534b5b-95fc-4041-abfe-ce03a8a318c2 service glance] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:31.100895 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94534b5b-95fc-4041-abfe-ce03a8a318c2 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:31.110745 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94534b5b-95fc-4041-abfe-ce03a8a318c2 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:31.112835 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94534b5b-95fc-4041-abfe-ce03a8a318c2 service glance] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:31.131238 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 997/2009] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:46:31 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:46:32.224495 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07eddcd5-38f3-452e-913c-2d2ef1a4e33c None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:32.225153 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07eddcd5-38f3-452e-913c-2d2ef1a4e33c None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:32.225153 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07eddcd5-38f3-452e-913c-2d2ef1a4e33c None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:32.226502 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1013/2010] 10.222.0.25 () {54 vars in 921 bytes} [Fri Oct 28 09:46:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:46:32.243134 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2675ba6a-c289-4d22-84df-56b57504a175 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:32.243134 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2675ba6a-c289-4d22-84df-56b57504a175 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:32.243134 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2675ba6a-c289-4d22-84df-56b57504a175 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:32.291325 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-2675ba6a-c289-4d22-84df-56b57504a175 None None] MFA Rules not processed for user `3f1d95049d2d4593bf52670b1a0035fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:32.292763 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-2675ba6a-c289-4d22-84df-56b57504a175 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:32.451867 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07eddcd5-38f3-452e-913c-2d2ef1a4e33c None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:32.453791 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-07eddcd5-38f3-452e-913c-2d2ef1a4e33c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:32.491122 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=93,keystone:DELETE=4,keystone:INSERT=3 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:32.520373 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 998/2011] 10.222.0.25 () {58 vars in 1018 bytes} [Fri Oct 28 09:46:32 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 279 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:46:32.535249 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2675ba6a-c289-4d22-84df-56b57504a175 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:32.538380 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-2675ba6a-c289-4d22-84df-56b57504a175 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:32.568819 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2675ba6a-c289-4d22-84df-56b57504a175 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:32.580919 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772404d0-1643-4593-9073-1acefa45ddcf service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3f1d95049d2d4593bf52670b1a0035fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:32.581670 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772404d0-1643-4593-9073-1acefa45ddcf service placement] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:32.581957 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772404d0-1643-4593-9073-1acefa45ddcf service placement] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:32.582233 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772404d0-1643-4593-9073-1acefa45ddcf service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:32.591864 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772404d0-1643-4593-9073-1acefa45ddcf service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:32.593587 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772404d0-1643-4593-9073-1acefa45ddcf service placement] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:32.608984 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 999/2012] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:46:32 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 74 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:46:32.623794 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07eddcd5-38f3-452e-913c-2d2ef1a4e33c None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:32.633477 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6afe59f-74d9-41ed-9263-bb27aab87a5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:32.634289 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6afe59f-74d9-41ed-9263-bb27aab87a5d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:32.634519 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6afe59f-74d9-41ed-9263-bb27aab87a5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:32.634794 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6afe59f-74d9-41ed-9263-bb27aab87a5d admin admin] PATH_INFO: `/v3/users/e451d264e4964409b09e0b96ae8b983e` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:32.636905 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6afe59f-74d9-41ed-9263-bb27aab87a5d admin admin] RBAC: Authorizing `identity:delete_user(user_id=e451d264e4964409b09e0b96ae8b983e)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:32.639219 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6afe59f-74d9-41ed-9263-bb27aab87a5d admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:32.739558 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e6afe59f-74d9-41ed-9263-bb27aab87a5d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e451d264e4964409b09e0b96ae8b983e'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:32.750521 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e6afe59f-74d9-41ed-9263-bb27aab87a5d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e451d264e4964409b09e0b96ae8b983e'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:32.751301 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e6afe59f-74d9-41ed-9263-bb27aab87a5d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e451d264e4964409b09e0b96ae8b983e'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:32.751990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e6afe59f-74d9-41ed-9263-bb27aab87a5d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e451d264e4964409b09e0b96ae8b983e'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:32.763767 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e6afe59f-74d9-41ed-9263-bb27aab87a5d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e451d264e4964409b09e0b96ae8b983e'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:32.805433 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1014/2013] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:46:32 2022] DELETE /identity/v3/users/e451d264e4964409b09e0b96ae8b983e => generated 0 bytes in 354 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:46:33.077958 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772404d0-1643-4593-9073-1acefa45ddcf service placement] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:33.107308 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772404d0-1643-4593-9073-1acefa45ddcf service placement] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:33.215114 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80219125-36df-46a6-a981-6d9f9898f91b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:33.215812 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80219125-36df-46a6-a981-6d9f9898f91b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:33.216097 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80219125-36df-46a6-a981-6d9f9898f91b admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:33.216436 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80219125-36df-46a6-a981-6d9f9898f91b admin admin] PATH_INFO: `/v3/projects/41516f10d050497e82056ddd1be5981c` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:33.217624 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80219125-36df-46a6-a981-6d9f9898f91b admin admin] RBAC: Authorizing `identity:delete_project(project_id=41516f10d050497e82056ddd1be5981c)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:33.219582 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80219125-36df-46a6-a981-6d9f9898f91b admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:33.298330 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-80219125-36df-46a6-a981-6d9f9898f91b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '41516f10d050497e82056ddd1be5981c', 'request_id': 'req-80219125-36df-46a6-a981-6d9f9898f91b'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:33.308946 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-80219125-36df-46a6-a981-6d9f9898f91b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '41516f10d050497e82056ddd1be5981c', 'request_id': 'req-80219125-36df-46a6-a981-6d9f9898f91b'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:33.324428 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-80219125-36df-46a6-a981-6d9f9898f91b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '41516f10d050497e82056ddd1be5981c', 'request_id': 'req-80219125-36df-46a6-a981-6d9f9898f91b'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:33.344805 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-80219125-36df-46a6-a981-6d9f9898f91b admin admin] The token cache is being invalidate because project 41516f10d050497e82056ddd1be5981c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:46:33.345127 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-80219125-36df-46a6-a981-6d9f9898f91b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:33.345822 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-80219125-36df-46a6-a981-6d9f9898f91b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:33.356960 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1000/2014] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:46:33 2022] DELETE /identity/v3/projects/41516f10d050497e82056ddd1be5981c => generated 0 bytes in 270 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:33.359113 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc332bd1-a8eb-4a9d-b7a1-ae0e8e1dca14 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:33.359996 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc332bd1-a8eb-4a9d-b7a1-ae0e8e1dca14 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:33.360479 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc332bd1-a8eb-4a9d-b7a1-ae0e8e1dca14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:33.384066 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-dc332bd1-a8eb-4a9d-b7a1-ae0e8e1dca14 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:33.385743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-dc332bd1-a8eb-4a9d-b7a1-ae0e8e1dca14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:33.462162 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:33.497130 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1015/2015] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:46:33.505748 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f3895e-9c38-47ab-a4ee-03b79bfdc22a None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:33.506156 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f3895e-9c38-47ab-a4ee-03b79bfdc22a None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:33.506545 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f3895e-9c38-47ab-a4ee-03b79bfdc22a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:33.535406 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-e9f3895e-9c38-47ab-a4ee-03b79bfdc22a None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:33.537224 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-e9f3895e-9c38-47ab-a4ee-03b79bfdc22a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:33.549693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1001/2016] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:46:33.561876 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc332bd1-a8eb-4a9d-b7a1-ae0e8e1dca14 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:33.573948 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc332bd1-a8eb-4a9d-b7a1-ae0e8e1dca14 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:33.594094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4e84d2-81b5-46a2-a527-63db3cfa404c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:33.595030 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4e84d2-81b5-46a2-a527-63db3cfa404c admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:33.595474 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4e84d2-81b5-46a2-a527-63db3cfa404c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:33.595906 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef4e84d2-81b5-46a2-a527-63db3cfa404c admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:33.596732 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef4e84d2-81b5-46a2-a527-63db3cfa404c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:46:33.599769 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef4e84d2-81b5-46a2-a527-63db3cfa404c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:33.599769 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef4e84d2-81b5-46a2-a527-63db3cfa404c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:33.626485 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1016/2017] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:46:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:46:33.626733 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f3895e-9c38-47ab-a4ee-03b79bfdc22a None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:33.641841 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f3895e-9c38-47ab-a4ee-03b79bfdc22a None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:33.649383 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a30c9703-0e5e-4e50-8ddb-d28a6773f0c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:33.650531 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a30c9703-0e5e-4e50-8ddb-d28a6773f0c2 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:33.651201 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a30c9703-0e5e-4e50-8ddb-d28a6773f0c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:33.651709 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a30c9703-0e5e-4e50-8ddb-d28a6773f0c2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:33.652617 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a30c9703-0e5e-4e50-8ddb-d28a6773f0c2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:33.654845 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a30c9703-0e5e-4e50-8ddb-d28a6773f0c2 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:33.691659 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1002/2018] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:46:33 2022] POST /identity/v3/projects => generated 393 bytes in 70 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:46:33.696025 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4e84d2-81b5-46a2-a527-63db3cfa404c admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:33.709499 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef4e84d2-81b5-46a2-a527-63db3cfa404c admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:33.782739 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96893fd-6c00-4cbc-ba00-e0032e426b74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:33.783439 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96893fd-6c00-4cbc-ba00-e0032e426b74 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:33.783924 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96893fd-6c00-4cbc-ba00-e0032e426b74 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:33.784271 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96893fd-6c00-4cbc-ba00-e0032e426b74 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:33.784987 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a96893fd-6c00-4cbc-ba00-e0032e426b74 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:33.787227 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a96893fd-6c00-4cbc-ba00-e0032e426b74 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:33.815855 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1017/2019] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:46:33 2022] POST /identity/v3/users => generated 297 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:46:33.820366 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a30c9703-0e5e-4e50-8ddb-d28a6773f0c2 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:33.831458 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a30c9703-0e5e-4e50-8ddb-d28a6773f0c2 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:33.843182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1c0189-6291-4c2c-b20e-65a01d5f2b29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:33.843853 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1c0189-6291-4c2c-b20e-65a01d5f2b29 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:33.844137 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1c0189-6291-4c2c-b20e-65a01d5f2b29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:33.844412 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1c0189-6291-4c2c-b20e-65a01d5f2b29 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:33.845096 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c1c0189-6291-4c2c-b20e-65a01d5f2b29 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:33.846775 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c1c0189-6291-4c2c-b20e-65a01d5f2b29 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:33.862786 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1003/2020] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:46:33 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:46:33.867549 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96893fd-6c00-4cbc-ba00-e0032e426b74 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:33.879069 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96893fd-6c00-4cbc-ba00-e0032e426b74 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:33.884027 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0753a93-72bb-4e13-82bc-1e93afb24930 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:33.884633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0753a93-72bb-4e13-82bc-1e93afb24930 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:33.885039 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0753a93-72bb-4e13-82bc-1e93afb24930 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:33.885322 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0753a93-72bb-4e13-82bc-1e93afb24930 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:33.885962 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0753a93-72bb-4e13-82bc-1e93afb24930 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:33.887720 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0753a93-72bb-4e13-82bc-1e93afb24930 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:33.906607 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1c0189-6291-4c2c-b20e-65a01d5f2b29 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:33.916601 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1018/2021] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:46:33 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:46:33.918094 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1c0189-6291-4c2c-b20e-65a01d5f2b29 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:33.925257 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d08dcbc-331b-4b2a-98ea-6bce794db37c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:33.926066 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d08dcbc-331b-4b2a-98ea-6bce794db37c admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:33.926345 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d08dcbc-331b-4b2a-98ea-6bce794db37c admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:33.927164 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d08dcbc-331b-4b2a-98ea-6bce794db37c admin admin] PATH_INFO: `/v3/projects/cb39b66aaf164124bf8766942a524830/users/a20b46753a71458fa7f9b9d17e620a9a/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:33.945718 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d08dcbc-331b-4b2a-98ea-6bce794db37c admin admin] RBAC: Authorizing `identity:create_grant(project_id=cb39b66aaf164124bf8766942a524830, user_id=a20b46753a71458fa7f9b9d17e620a9a, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:33.948783 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d08dcbc-331b-4b2a-98ea-6bce794db37c admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:33.978345 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a131cd7f-9aa2-4f0d-aca3-780f2dfa5858 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:33.978656 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a131cd7f-9aa2-4f0d-aca3-780f2dfa5858 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:33.978935 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a131cd7f-9aa2-4f0d-aca3-780f2dfa5858 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:33.981672 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1004/2022] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:46:33 2022] PUT /identity/v3/projects/cb39b66aaf164124bf8766942a524830/users/a20b46753a71458fa7f9b9d17e620a9a/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:46:34.028385 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-a131cd7f-9aa2-4f0d-aca3-780f2dfa5858 None None] MFA Rules not processed for user `a20b46753a71458fa7f9b9d17e620a9a`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:34.029841 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a131cd7f-9aa2-4f0d-aca3-780f2dfa5858 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:34.125955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1019/2023] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:33 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:46:34.130646 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d08dcbc-331b-4b2a-98ea-6bce794db37c admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:34.152375 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:34.246625 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d08dcbc-331b-4b2a-98ea-6bce794db37c admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:34.246625 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c188307-6a4f-4ce9-9763-85874aab2816 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:34.246625 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c188307-6a4f-4ce9-9763-85874aab2816 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:34.246625 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c188307-6a4f-4ce9-9763-85874aab2816 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:34.246992 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c188307-6a4f-4ce9-9763-85874aab2816 admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:34.246992 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c188307-6a4f-4ce9-9763-85874aab2816 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:34.246992 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c188307-6a4f-4ce9-9763-85874aab2816 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:34.282450 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1005/2024] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:46:34 2022] POST /identity/v3/projects => generated 391 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:46:34.285437 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a131cd7f-9aa2-4f0d-aca3-780f2dfa5858 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:34.297485 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a131cd7f-9aa2-4f0d-aca3-780f2dfa5858 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:34.357328 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:34.378679 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d4263d-e517-4b1c-8550-449603a86c42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:34.379381 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d4263d-e517-4b1c-8550-449603a86c42 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:34.379644 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d4263d-e517-4b1c-8550-449603a86c42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:34.379922 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d4263d-e517-4b1c-8550-449603a86c42 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:34.380631 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d4263d-e517-4b1c-8550-449603a86c42 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:34.383777 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d4263d-e517-4b1c-8550-449603a86c42 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:34.410556 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1020/2025] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:46:34 2022] POST /identity/v3/users => generated 302 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:46:34.417424 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c188307-6a4f-4ce9-9763-85874aab2816 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:34.429472 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c188307-6a4f-4ce9-9763-85874aab2816 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:34.437075 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edae78a3-6ebf-412c-abac-2d4df49bcbb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:34.437732 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edae78a3-6ebf-412c-abac-2d4df49bcbb8 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:34.438022 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edae78a3-6ebf-412c-abac-2d4df49bcbb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:34.438523 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edae78a3-6ebf-412c-abac-2d4df49bcbb8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:34.439175 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edae78a3-6ebf-412c-abac-2d4df49bcbb8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:34.440935 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edae78a3-6ebf-412c-abac-2d4df49bcbb8 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:34.458178 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1006/2026] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:46:34 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:46:34.460542 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d4263d-e517-4b1c-8550-449603a86c42 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:34.472083 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d4263d-e517-4b1c-8550-449603a86c42 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:34.481568 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a573024-4d66-4c59-b7a1-be6fc9769160 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:34.482257 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a573024-4d66-4c59-b7a1-be6fc9769160 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:34.482606 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a573024-4d66-4c59-b7a1-be6fc9769160 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:34.482858 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a573024-4d66-4c59-b7a1-be6fc9769160 admin admin] PATH_INFO: `/v3/projects/5876c913133e469fb3963a92e3d1e797/users/5d3ae26101f34deeaa7a9571de0a0323/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:34.503504 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a573024-4d66-4c59-b7a1-be6fc9769160 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5876c913133e469fb3963a92e3d1e797, user_id=5d3ae26101f34deeaa7a9571de0a0323, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:34.506484 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a573024-4d66-4c59-b7a1-be6fc9769160 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:34.523876 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1021/2027] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:46:34 2022] PUT /identity/v3/projects/5876c913133e469fb3963a92e3d1e797/users/5d3ae26101f34deeaa7a9571de0a0323/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:34.527768 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edae78a3-6ebf-412c-abac-2d4df49bcbb8 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:34.542597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edae78a3-6ebf-412c-abac-2d4df49bcbb8 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:34.631004 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a605b79e-3848-42f4-9951-d0dfec6acdd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:34.632086 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a605b79e-3848-42f4-9951-d0dfec6acdd8 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:34.632384 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a605b79e-3848-42f4-9951-d0dfec6acdd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:34.632666 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a605b79e-3848-42f4-9951-d0dfec6acdd8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:34.633321 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a605b79e-3848-42f4-9951-d0dfec6acdd8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:34.635372 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a605b79e-3848-42f4-9951-d0dfec6acdd8 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:34.651538 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1007/2028] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:46:34 2022] GET /identity/v3/roles => generated 1420 bytes in 124 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:46:34.655456 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a573024-4d66-4c59-b7a1-be6fc9769160 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:34.667809 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a573024-4d66-4c59-b7a1-be6fc9769160 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:34.676065 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5649c58-bb32-4011-a54f-b336b3dc9e07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:34.677364 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5649c58-bb32-4011-a54f-b336b3dc9e07 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:34.677767 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5649c58-bb32-4011-a54f-b336b3dc9e07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:34.678109 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5649c58-bb32-4011-a54f-b336b3dc9e07 admin admin] PATH_INFO: `/v3/projects/5876c913133e469fb3963a92e3d1e797/users/5d3ae26101f34deeaa7a9571de0a0323/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:34.680601 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5649c58-bb32-4011-a54f-b336b3dc9e07 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5876c913133e469fb3963a92e3d1e797, user_id=5d3ae26101f34deeaa7a9571de0a0323, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:34.683369 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5649c58-bb32-4011-a54f-b336b3dc9e07 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:34.702875 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1022/2029] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:46:34 2022] PUT /identity/v3/projects/5876c913133e469fb3963a92e3d1e797/users/5d3ae26101f34deeaa7a9571de0a0323/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:34.717292 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d28fab8-359f-4ce9-9583-7631ac987047 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:34.717861 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d28fab8-359f-4ce9-9583-7631ac987047 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:34.718138 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d28fab8-359f-4ce9-9583-7631ac987047 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:34.764992 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-7d28fab8-359f-4ce9-9583-7631ac987047 None None] MFA Rules not processed for user `5d3ae26101f34deeaa7a9571de0a0323`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:34.766349 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-7d28fab8-359f-4ce9-9583-7631ac987047 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:34.862993 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1008/2030] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:34 2022] POST /identity/v3/auth/tokens => generated 3335 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:46:34.873741 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5649c58-bb32-4011-a54f-b336b3dc9e07 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:34.880058 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-b5649c58-bb32-4011-a54f-b336b3dc9e07 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:34.971584 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5649c58-bb32-4011-a54f-b336b3dc9e07 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:34.976031 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63461c91-e433-44f3-b18c-f11b790357c7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:34.976661 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63461c91-e433-44f3-b18c-f11b790357c7 service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:34.976942 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63461c91-e433-44f3-b18c-f11b790357c7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:34.977220 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63461c91-e433-44f3-b18c-f11b790357c7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:34.989478 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63461c91-e433-44f3-b18c-f11b790357c7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:34.990991 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63461c91-e433-44f3-b18c-f11b790357c7 service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:35.010236 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1023/2031] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:46:34 2022] GET /identity/v3/auth/tokens => generated 3335 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:46:35.063531 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d28fab8-359f-4ce9-9583-7631ac987047 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:35.065373 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-7d28fab8-359f-4ce9-9583-7631ac987047 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:35.080761 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d28fab8-359f-4ce9-9583-7631ac987047 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:35.085243 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e94cf31-1401-49e2-9691-aeb26bb7af1e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:35.086090 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e94cf31-1401-49e2-9691-aeb26bb7af1e service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:35.086605 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e94cf31-1401-49e2-9691-aeb26bb7af1e service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:35.087044 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e94cf31-1401-49e2-9691-aeb26bb7af1e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:35.096219 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:35.161554 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e94cf31-1401-49e2-9691-aeb26bb7af1e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:35.165790 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e94cf31-1401-49e2-9691-aeb26bb7af1e service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:35.176225 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1009/2032] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:46:35 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:46:38.889899 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63461c91-e433-44f3-b18c-f11b790357c7 service cinder] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:38.979599 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63461c91-e433-44f3-b18c-f11b790357c7 service cinder] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:38.984260 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddddf413-ee3f-40a7-94c8-aede154f4b55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:38.984928 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddddf413-ee3f-40a7-94c8-aede154f4b55 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:38.985269 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddddf413-ee3f-40a7-94c8-aede154f4b55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:38.985548 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddddf413-ee3f-40a7-94c8-aede154f4b55 admin admin] PATH_INFO: `/v3/users/a20b46753a71458fa7f9b9d17e620a9a` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:38.988250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddddf413-ee3f-40a7-94c8-aede154f4b55 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a20b46753a71458fa7f9b9d17e620a9a)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:38.991148 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddddf413-ee3f-40a7-94c8-aede154f4b55 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:39.057011 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:INSERT=5,keystone:DELETE=6 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:39.079573 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ddddf413-ee3f-40a7-94c8-aede154f4b55 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a20b46753a71458fa7f9b9d17e620a9a'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:39.094326 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ddddf413-ee3f-40a7-94c8-aede154f4b55 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a20b46753a71458fa7f9b9d17e620a9a'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:39.095059 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ddddf413-ee3f-40a7-94c8-aede154f4b55 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a20b46753a71458fa7f9b9d17e620a9a'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:39.095756 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ddddf413-ee3f-40a7-94c8-aede154f4b55 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a20b46753a71458fa7f9b9d17e620a9a'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:39.123581 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ddddf413-ee3f-40a7-94c8-aede154f4b55 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a20b46753a71458fa7f9b9d17e620a9a'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:39.157368 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1024/2033] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:46:38 2022] DELETE /identity/v3/users/a20b46753a71458fa7f9b9d17e620a9a => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:39.162200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e19171f-4e76-446e-9360-fd2840bbcbf7 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:39.163362 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e19171f-4e76-446e-9360-fd2840bbcbf7 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:39.163876 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e19171f-4e76-446e-9360-fd2840bbcbf7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:39.260945 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-7e19171f-4e76-446e-9360-fd2840bbcbf7 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:39.262352 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-7e19171f-4e76-446e-9360-fd2840bbcbf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:39.356116 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1010/2034] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:46:39.368724 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddddf413-ee3f-40a7-94c8-aede154f4b55 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:39.370193 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ddddf413-ee3f-40a7-94c8-aede154f4b55 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:39.443411 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddddf413-ee3f-40a7-94c8-aede154f4b55 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:39.447050 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b210791f-602a-413a-9737-61537e9829d2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:39.447670 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b210791f-602a-413a-9737-61537e9829d2 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:39.447951 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b210791f-602a-413a-9737-61537e9829d2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:39.448230 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b210791f-602a-413a-9737-61537e9829d2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:39.460220 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b210791f-602a-413a-9737-61537e9829d2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:39.461664 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b210791f-602a-413a-9737-61537e9829d2 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:39.472750 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1025/2035] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:46:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:46:39.844820 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e19171f-4e76-446e-9360-fd2840bbcbf7 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:39.846257 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-7e19171f-4e76-446e-9360-fd2840bbcbf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:39.859619 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e19171f-4e76-446e-9360-fd2840bbcbf7 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:39.866321 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da0ade0-71f0-4f8e-a616-fba1a9cae464 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:39.867008 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da0ade0-71f0-4f8e-a616-fba1a9cae464 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:39.867292 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da0ade0-71f0-4f8e-a616-fba1a9cae464 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:39.867570 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da0ade0-71f0-4f8e-a616-fba1a9cae464 admin admin] PATH_INFO: `/v3/projects/cb39b66aaf164124bf8766942a524830` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:39.868770 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da0ade0-71f0-4f8e-a616-fba1a9cae464 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cb39b66aaf164124bf8766942a524830)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:39.871034 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da0ade0-71f0-4f8e-a616-fba1a9cae464 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:39.938661 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-8da0ade0-71f0-4f8e-a616-fba1a9cae464 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cb39b66aaf164124bf8766942a524830', 'request_id': 'req-8da0ade0-71f0-4f8e-a616-fba1a9cae464'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:39.946577 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-8da0ade0-71f0-4f8e-a616-fba1a9cae464 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cb39b66aaf164124bf8766942a524830', 'request_id': 'req-8da0ade0-71f0-4f8e-a616-fba1a9cae464'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:39.955675 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-8da0ade0-71f0-4f8e-a616-fba1a9cae464 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cb39b66aaf164124bf8766942a524830', 'request_id': 'req-8da0ade0-71f0-4f8e-a616-fba1a9cae464'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:39.970178 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-8da0ade0-71f0-4f8e-a616-fba1a9cae464 admin admin] The token cache is being invalidate because project cb39b66aaf164124bf8766942a524830 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:46:39.970178 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-8da0ade0-71f0-4f8e-a616-fba1a9cae464 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:39.970806 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-8da0ade0-71f0-4f8e-a616-fba1a9cae464 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:39.972373 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1011/2036] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:46:39 2022] DELETE /identity/v3/projects/cb39b66aaf164124bf8766942a524830 => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:39.977042 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b210791f-602a-413a-9737-61537e9829d2 service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:39.987000 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b210791f-602a-413a-9737-61537e9829d2 service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:40.057807 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e017b5d-9c06-4cb6-9b02-f8f998bd7296 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:40.058480 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e017b5d-9c06-4cb6-9b02-f8f998bd7296 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:40.058761 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e017b5d-9c06-4cb6-9b02-f8f998bd7296 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:40.059037 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e017b5d-9c06-4cb6-9b02-f8f998bd7296 admin admin] PATH_INFO: `/v3/users/5d3ae26101f34deeaa7a9571de0a0323` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:40.060184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e017b5d-9c06-4cb6-9b02-f8f998bd7296 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5d3ae26101f34deeaa7a9571de0a0323)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:40.062061 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e017b5d-9c06-4cb6-9b02-f8f998bd7296 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:40.119875 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-4e017b5d-9c06-4cb6-9b02-f8f998bd7296 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5d3ae26101f34deeaa7a9571de0a0323'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:40.130079 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-4e017b5d-9c06-4cb6-9b02-f8f998bd7296 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5d3ae26101f34deeaa7a9571de0a0323'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:40.130798 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-4e017b5d-9c06-4cb6-9b02-f8f998bd7296 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5d3ae26101f34deeaa7a9571de0a0323'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:40.131511 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-4e017b5d-9c06-4cb6-9b02-f8f998bd7296 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5d3ae26101f34deeaa7a9571de0a0323'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:40.135599 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c515bd9-f9ea-4289-a630-36a144340ff9 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:40.135889 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c515bd9-f9ea-4289-a630-36a144340ff9 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:40.136176 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c515bd9-f9ea-4289-a630-36a144340ff9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:40.139229 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-4e017b5d-9c06-4cb6-9b02-f8f998bd7296 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5d3ae26101f34deeaa7a9571de0a0323'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:40.141269 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:DELETE=12,keystone:SELECT=86,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:40.158832 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-7c515bd9-f9ea-4289-a630-36a144340ff9 None None] MFA Rules not processed for user `23acf07cede847a69ca839ee200accd3`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:40.160037 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-7c515bd9-f9ea-4289-a630-36a144340ff9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:40.165195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1026/2037] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:46:39 2022] DELETE /identity/v3/users/5d3ae26101f34deeaa7a9571de0a0323 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:40.193775 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:40.228285 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1012/2038] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:40 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:46:40.466244 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e017b5d-9c06-4cb6-9b02-f8f998bd7296 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:40.468324 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4e017b5d-9c06-4cb6-9b02-f8f998bd7296 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:40.553985 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e017b5d-9c06-4cb6-9b02-f8f998bd7296 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:40.560833 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05b7ae4-185a-48cf-b80b-f93cc4b0d4ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:40.561492 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05b7ae4-185a-48cf-b80b-f93cc4b0d4ed admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:40.561771 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05b7ae4-185a-48cf-b80b-f93cc4b0d4ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:40.562042 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05b7ae4-185a-48cf-b80b-f93cc4b0d4ed admin admin] PATH_INFO: `/v3/projects/5876c913133e469fb3963a92e3d1e797` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:40.563200 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a05b7ae4-185a-48cf-b80b-f93cc4b0d4ed admin admin] RBAC: Authorizing `identity:delete_project(project_id=5876c913133e469fb3963a92e3d1e797)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:40.565255 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a05b7ae4-185a-48cf-b80b-f93cc4b0d4ed admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:40.618276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a05b7ae4-185a-48cf-b80b-f93cc4b0d4ed admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5876c913133e469fb3963a92e3d1e797', 'request_id': 'req-a05b7ae4-185a-48cf-b80b-f93cc4b0d4ed'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:40.624618 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a05b7ae4-185a-48cf-b80b-f93cc4b0d4ed admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5876c913133e469fb3963a92e3d1e797', 'request_id': 'req-a05b7ae4-185a-48cf-b80b-f93cc4b0d4ed'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:40.633764 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a05b7ae4-185a-48cf-b80b-f93cc4b0d4ed admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5876c913133e469fb3963a92e3d1e797', 'request_id': 'req-a05b7ae4-185a-48cf-b80b-f93cc4b0d4ed'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:40.645662 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a05b7ae4-185a-48cf-b80b-f93cc4b0d4ed admin admin] The token cache is being invalidate because project 5876c913133e469fb3963a92e3d1e797 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:46:40.646044 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a05b7ae4-185a-48cf-b80b-f93cc4b0d4ed admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:40.646648 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a05b7ae4-185a-48cf-b80b-f93cc4b0d4ed admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:40.648143 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1027/2039] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:46:40 2022] DELETE /identity/v3/projects/5876c913133e469fb3963a92e3d1e797 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:40.652931 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17ecedf-5ca8-469b-a824-cc5d93783b7e None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:40.653206 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17ecedf-5ca8-469b-a824-cc5d93783b7e None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:40.653481 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17ecedf-5ca8-469b-a824-cc5d93783b7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:40.674934 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-e17ecedf-5ca8-469b-a824-cc5d93783b7e None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:40.676678 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-e17ecedf-5ca8-469b-a824-cc5d93783b7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:40.754688 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1013/2040] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:46:40.762299 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e28e1a5-fafa-4a1a-a31d-c7877d752bae None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:40.762630 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e28e1a5-fafa-4a1a-a31d-c7877d752bae None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:40.762915 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e28e1a5-fafa-4a1a-a31d-c7877d752bae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:40.783740 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-7e28e1a5-fafa-4a1a-a31d-c7877d752bae None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:40.785011 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7e28e1a5-fafa-4a1a-a31d-c7877d752bae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:40.795586 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1028/2041] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:46:40.800033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17ecedf-5ca8-469b-a824-cc5d93783b7e None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:40.810157 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17ecedf-5ca8-469b-a824-cc5d93783b7e None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:40.815130 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b81401-beb5-44a1-a079-7ffaa06a3b6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:40.815758 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b81401-beb5-44a1-a079-7ffaa06a3b6b admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:40.816742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b81401-beb5-44a1-a079-7ffaa06a3b6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:40.817520 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b81401-beb5-44a1-a079-7ffaa06a3b6b admin admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:40.818436 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0b81401-beb5-44a1-a079-7ffaa06a3b6b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:46:40.818656 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0b81401-beb5-44a1-a079-7ffaa06a3b6b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:40.820196 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0b81401-beb5-44a1-a079-7ffaa06a3b6b admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:40.833703 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1014/2042] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:46:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:46:40.839565 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e28e1a5-fafa-4a1a-a31d-c7877d752bae None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:40.853129 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e28e1a5-fafa-4a1a-a31d-c7877d752bae None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:40.858308 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a8c97b-f42e-4fe5-a5f7-eaf3587432cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:40.859481 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a8c97b-f42e-4fe5-a5f7-eaf3587432cd admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:40.859771 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a8c97b-f42e-4fe5-a5f7-eaf3587432cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:40.860116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a8c97b-f42e-4fe5-a5f7-eaf3587432cd admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:40.860752 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a8c97b-f42e-4fe5-a5f7-eaf3587432cd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:40.864275 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1a8c97b-f42e-4fe5-a5f7-eaf3587432cd admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:40.909796 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1029/2043] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:46:40 2022] POST /identity/v3/projects => generated 401 bytes in 72 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:46:40.914835 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b81401-beb5-44a1-a079-7ffaa06a3b6b admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:40.926569 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b81401-beb5-44a1-a079-7ffaa06a3b6b admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:41.013973 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee042f6-e99f-4e3d-94e2-810ff030789f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:41.014653 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee042f6-e99f-4e3d-94e2-810ff030789f admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:41.014958 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee042f6-e99f-4e3d-94e2-810ff030789f admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:41.015193 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee042f6-e99f-4e3d-94e2-810ff030789f admin admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:41.015845 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ee042f6-e99f-4e3d-94e2-810ff030789f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:41.018142 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ee042f6-e99f-4e3d-94e2-810ff030789f admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:41.047028 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1015/2044] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:46:40 2022] POST /identity/v3/users => generated 301 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:46:41.050983 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a8c97b-f42e-4fe5-a5f7-eaf3587432cd admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:41.061585 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a8c97b-f42e-4fe5-a5f7-eaf3587432cd admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:41.066175 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626c4af1-8f61-497c-a95e-135aeab0d49d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:41.066809 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626c4af1-8f61-497c-a95e-135aeab0d49d admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:41.067628 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626c4af1-8f61-497c-a95e-135aeab0d49d admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:41.067903 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626c4af1-8f61-497c-a95e-135aeab0d49d admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:41.068549 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-626c4af1-8f61-497c-a95e-135aeab0d49d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:41.070169 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-626c4af1-8f61-497c-a95e-135aeab0d49d admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:41.081419 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1030/2045] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:46:41 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:46:41.086559 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee042f6-e99f-4e3d-94e2-810ff030789f admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:41.097493 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee042f6-e99f-4e3d-94e2-810ff030789f admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:41.103982 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d68b95e6-1ded-47d6-aa4e-bffc207ea192 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:41.104658 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68b95e6-1ded-47d6-aa4e-bffc207ea192 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:41.104942 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68b95e6-1ded-47d6-aa4e-bffc207ea192 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:41.105221 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d68b95e6-1ded-47d6-aa4e-bffc207ea192 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:41.105891 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d68b95e6-1ded-47d6-aa4e-bffc207ea192 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:41.107568 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d68b95e6-1ded-47d6-aa4e-bffc207ea192 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:41.119042 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1016/2046] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:46:41 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:46:41.123616 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626c4af1-8f61-497c-a95e-135aeab0d49d admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:41.135559 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626c4af1-8f61-497c-a95e-135aeab0d49d admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:41.140860 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3b2275-3a80-4ddf-bcc8-d8d374dcdb92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:41.141536 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3b2275-3a80-4ddf-bcc8-d8d374dcdb92 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:41.141820 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3b2275-3a80-4ddf-bcc8-d8d374dcdb92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:41.142102 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3b2275-3a80-4ddf-bcc8-d8d374dcdb92 admin admin] PATH_INFO: `/v3/projects/8461445392af4e50ba987c8a576e2dca/users/0c2911ca871b4d54876976578eee2010/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:41.158512 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f3b2275-3a80-4ddf-bcc8-d8d374dcdb92 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8461445392af4e50ba987c8a576e2dca, user_id=0c2911ca871b4d54876976578eee2010, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:41.161288 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f3b2275-3a80-4ddf-bcc8-d8d374dcdb92 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:41.177264 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1031/2047] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:46:41 2022] PUT /identity/v3/projects/8461445392af4e50ba987c8a576e2dca/users/0c2911ca871b4d54876976578eee2010/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:41.189834 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4349f8a1-37d0-4f55-b167-949a6bad8c25 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:41.190107 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4349f8a1-37d0-4f55-b167-949a6bad8c25 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:41.190363 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4349f8a1-37d0-4f55-b167-949a6bad8c25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:41.225788 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-4349f8a1-37d0-4f55-b167-949a6bad8c25 None None] MFA Rules not processed for user `0c2911ca871b4d54876976578eee2010`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:41.227213 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-4349f8a1-37d0-4f55-b167-949a6bad8c25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:41.259959 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:41.301850 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1017/2048] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:41 2022] POST /identity/v3/auth/tokens => generated 3278 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:46:41.306386 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3b2275-3a80-4ddf-bcc8-d8d374dcdb92 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:41.335580 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=88,keystone:DELETE=9,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:41.409070 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3b2275-3a80-4ddf-bcc8-d8d374dcdb92 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:41.413873 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22ff993-f7c4-4584-96db-e321ff0e8b66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:41.414520 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22ff993-f7c4-4584-96db-e321ff0e8b66 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:41.414807 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22ff993-f7c4-4584-96db-e321ff0e8b66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:41.415083 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22ff993-f7c4-4584-96db-e321ff0e8b66 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:41.415718 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d22ff993-f7c4-4584-96db-e321ff0e8b66 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:41.417895 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d22ff993-f7c4-4584-96db-e321ff0e8b66 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:41.453030 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1032/2049] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:46:41 2022] POST /identity/v3/projects => generated 401 bytes in 147 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:46:41.457306 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4349f8a1-37d0-4f55-b167-949a6bad8c25 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:41.469229 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4349f8a1-37d0-4f55-b167-949a6bad8c25 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:41.539039 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c74c19a-cf85-47e7-b0dd-c26181008cee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:41.539778 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c74c19a-cf85-47e7-b0dd-c26181008cee admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:41.540156 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c74c19a-cf85-47e7-b0dd-c26181008cee admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:41.540389 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c74c19a-cf85-47e7-b0dd-c26181008cee admin admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:41.541350 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c74c19a-cf85-47e7-b0dd-c26181008cee admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:41.543692 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c74c19a-cf85-47e7-b0dd-c26181008cee admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:41.570450 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1018/2050] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:46:41 2022] POST /identity/v3/users => generated 307 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:46:41.574581 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22ff993-f7c4-4584-96db-e321ff0e8b66 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:41.586209 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22ff993-f7c4-4584-96db-e321ff0e8b66 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:41.591372 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba9975a-9a41-4c63-811b-a2566c6d3bd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:41.591982 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba9975a-9a41-4c63-811b-a2566c6d3bd8 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:41.592266 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba9975a-9a41-4c63-811b-a2566c6d3bd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:41.592995 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba9975a-9a41-4c63-811b-a2566c6d3bd8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:41.593185 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba9975a-9a41-4c63-811b-a2566c6d3bd8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:41.594834 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba9975a-9a41-4c63-811b-a2566c6d3bd8 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:41.606431 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1033/2051] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:46:41 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:46:41.610442 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c74c19a-cf85-47e7-b0dd-c26181008cee admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:41.622307 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c74c19a-cf85-47e7-b0dd-c26181008cee admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:41.627757 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72a7f8a-a2d5-4cdb-a6a3-6587e7aa04e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:41.628527 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72a7f8a-a2d5-4cdb-a6a3-6587e7aa04e0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:41.628869 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72a7f8a-a2d5-4cdb-a6a3-6587e7aa04e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:41.629216 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a72a7f8a-a2d5-4cdb-a6a3-6587e7aa04e0 admin admin] PATH_INFO: `/v3/projects/2d878a488ebb45d8872c74df48985ff0/users/961e8d5d5e934299b1ff29b96f28cf8e/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:41.649683 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72a7f8a-a2d5-4cdb-a6a3-6587e7aa04e0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2d878a488ebb45d8872c74df48985ff0, user_id=961e8d5d5e934299b1ff29b96f28cf8e, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:41.652847 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a72a7f8a-a2d5-4cdb-a6a3-6587e7aa04e0 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:41.667820 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1019/2052] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:46:41 2022] PUT /identity/v3/projects/2d878a488ebb45d8872c74df48985ff0/users/961e8d5d5e934299b1ff29b96f28cf8e/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:41.671909 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba9975a-9a41-4c63-811b-a2566c6d3bd8 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:41.683285 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba9975a-9a41-4c63-811b-a2566c6d3bd8 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:41.746797 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea782a5-ff53-421c-a48e-ab2cbc3f077c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:41.747413 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea782a5-ff53-421c-a48e-ab2cbc3f077c admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:41.747754 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea782a5-ff53-421c-a48e-ab2cbc3f077c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:41.748030 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea782a5-ff53-421c-a48e-ab2cbc3f077c admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:41.748673 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea782a5-ff53-421c-a48e-ab2cbc3f077c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:41.750243 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea782a5-ff53-421c-a48e-ab2cbc3f077c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:41.764364 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1034/2053] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:46:41 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:46:41.765783 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72a7f8a-a2d5-4cdb-a6a3-6587e7aa04e0 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:41.777426 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a72a7f8a-a2d5-4cdb-a6a3-6587e7aa04e0 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:41.786231 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e1b019-82d2-470f-9134-0e682dbac7fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:41.787121 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e1b019-82d2-470f-9134-0e682dbac7fe admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:41.787476 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e1b019-82d2-470f-9134-0e682dbac7fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:41.788310 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e1b019-82d2-470f-9134-0e682dbac7fe admin admin] PATH_INFO: `/v3/projects/2d878a488ebb45d8872c74df48985ff0/users/961e8d5d5e934299b1ff29b96f28cf8e/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:41.791080 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82e1b019-82d2-470f-9134-0e682dbac7fe admin admin] RBAC: Authorizing `identity:create_grant(project_id=2d878a488ebb45d8872c74df48985ff0, user_id=961e8d5d5e934299b1ff29b96f28cf8e, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:41.794257 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82e1b019-82d2-470f-9134-0e682dbac7fe admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:41.805186 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1020/2054] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:46:41 2022] PUT /identity/v3/projects/2d878a488ebb45d8872c74df48985ff0/users/961e8d5d5e934299b1ff29b96f28cf8e/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:46:41.815802 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe63324-bd4f-46ac-8323-61f921b26d73 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:41.816078 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe63324-bd4f-46ac-8323-61f921b26d73 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:41.816359 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe63324-bd4f-46ac-8323-61f921b26d73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:41.844988 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-6fe63324-bd4f-46ac-8323-61f921b26d73 None None] MFA Rules not processed for user `961e8d5d5e934299b1ff29b96f28cf8e`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:41.846215 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-6fe63324-bd4f-46ac-8323-61f921b26d73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:41.924529 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1035/2055] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:41 2022] POST /identity/v3/auth/tokens => generated 3345 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:46:41.935772 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e1b019-82d2-470f-9134-0e682dbac7fe admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:41.937463 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-82e1b019-82d2-470f-9134-0e682dbac7fe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:42.000013 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e1b019-82d2-470f-9134-0e682dbac7fe admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:42.010478 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7814178-d002-4539-b231-7d8f0c2fd6a7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:42.011097 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7814178-d002-4539-b231-7d8f0c2fd6a7 service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:42.011597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7814178-d002-4539-b231-7d8f0c2fd6a7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:42.012129 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7814178-d002-4539-b231-7d8f0c2fd6a7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:42.022217 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7814178-d002-4539-b231-7d8f0c2fd6a7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:42.024263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7814178-d002-4539-b231-7d8f0c2fd6a7 service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:42.037204 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1021/2056] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:46:41 2022] GET /identity/v3/auth/tokens => generated 3345 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:46:42.144917 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe63324-bd4f-46ac-8323-61f921b26d73 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:42.151372 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:42.154956 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe63324-bd4f-46ac-8323-61f921b26d73 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:42.159573 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2952ca-c3b0-46e6-a53c-17387ac93ca1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:42.159866 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2952ca-c3b0-46e6-a53c-17387ac93ca1 service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:42.160147 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2952ca-c3b0-46e6-a53c-17387ac93ca1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:42.160421 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2952ca-c3b0-46e6-a53c-17387ac93ca1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:42.225279 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2952ca-c3b0-46e6-a53c-17387ac93ca1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:42.226733 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2952ca-c3b0-46e6-a53c-17387ac93ca1 service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:42.235855 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1036/2057] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:46:42 2022] GET /identity/v3/auth/tokens => generated 3278 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:46:44.380715 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e32dbfb-164b-4719-9aad-cc138222bfb7 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:44.381326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e32dbfb-164b-4719-9aad-cc138222bfb7 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:44.381326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e32dbfb-164b-4719-9aad-cc138222bfb7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:44.400797 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-6e32dbfb-164b-4719-9aad-cc138222bfb7 None None] MFA Rules not processed for user `97c8877c74194a6bbbf25b3c8f5b7dbd`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:44.402003 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-6e32dbfb-164b-4719-9aad-cc138222bfb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:44.426153 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:44.475822 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1022/2058] 10.222.0.25 () {58 vars in 1009 bytes} [Fri Oct 28 09:46:44 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:46:44.489816 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2952ca-c3b0-46e6-a53c-17387ac93ca1 service cinder] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:44.491318 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-9d2952ca-c3b0-46e6-a53c-17387ac93ca1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:44.578908 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2952ca-c3b0-46e6-a53c-17387ac93ca1 service cinder] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:44.584027 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6486a16d-af18-4f91-b02b-c6be493bd40c service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c6cffe0d56e45c4b8fb54fbe5df6094', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:44.584666 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6486a16d-af18-4f91-b02b-c6be493bd40c service swift] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:44.584949 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6486a16d-af18-4f91-b02b-c6be493bd40c service swift] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:44.585225 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6486a16d-af18-4f91-b02b-c6be493bd40c service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:44.597767 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6486a16d-af18-4f91-b02b-c6be493bd40c service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:44.600314 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6486a16d-af18-4f91-b02b-c6be493bd40c service swift] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:44.608923 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1037/2059] 10.222.0.25 () {60 vars in 1384 bytes} [Fri Oct 28 09:46:44 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 28 09:46:46.613400 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e32dbfb-164b-4719-9aad-cc138222bfb7 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:46.737618 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e32dbfb-164b-4719-9aad-cc138222bfb7 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:46.748030 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb0a3d3-412a-45dc-80b7-940cc391d731 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:46.748688 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb0a3d3-412a-45dc-80b7-940cc391d731 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:46.748966 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb0a3d3-412a-45dc-80b7-940cc391d731 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:46.749250 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb0a3d3-412a-45dc-80b7-940cc391d731 admin admin] PATH_INFO: `/v3/users/0c2911ca871b4d54876976578eee2010` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:46.750771 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb0a3d3-412a-45dc-80b7-940cc391d731 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0c2911ca871b4d54876976578eee2010)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:46.752731 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb0a3d3-412a-45dc-80b7-940cc391d731 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:46.821057 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-edb0a3d3-412a-45dc-80b7-940cc391d731 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0c2911ca871b4d54876976578eee2010'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:46.831852 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-edb0a3d3-412a-45dc-80b7-940cc391d731 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0c2911ca871b4d54876976578eee2010'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:46.843412 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-edb0a3d3-412a-45dc-80b7-940cc391d731 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0c2911ca871b4d54876976578eee2010'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:46.845331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-edb0a3d3-412a-45dc-80b7-940cc391d731 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0c2911ca871b4d54876976578eee2010'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:46.860493 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-edb0a3d3-412a-45dc-80b7-940cc391d731 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0c2911ca871b4d54876976578eee2010'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:46.888544 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1023/2060] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:46:46 2022] DELETE /identity/v3/users/0c2911ca871b4d54876976578eee2010 => generated 0 bytes in 276 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:46.893879 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d076607-73af-4bd4-bbca-99df1dd58d82 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:46.894904 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d076607-73af-4bd4-bbca-99df1dd58d82 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:46.895452 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d076607-73af-4bd4-bbca-99df1dd58d82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:46.936683 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-5d076607-73af-4bd4-bbca-99df1dd58d82 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:46.937993 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-5d076607-73af-4bd4-bbca-99df1dd58d82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:47.011580 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1038/2061] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:46:47.026657 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb0a3d3-412a-45dc-80b7-940cc391d731 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:47.028228 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-edb0a3d3-412a-45dc-80b7-940cc391d731 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:47.167996 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb0a3d3-412a-45dc-80b7-940cc391d731 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:47.172194 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf75ffa-4d23-4123-9551-649a2085a912 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:47.173224 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf75ffa-4d23-4123-9551-649a2085a912 service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:47.173590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf75ffa-4d23-4123-9551-649a2085a912 service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:47.173980 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf75ffa-4d23-4123-9551-649a2085a912 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:47.180962 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:47.186623 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf75ffa-4d23-4123-9551-649a2085a912 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:47.188463 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf75ffa-4d23-4123-9551-649a2085a912 service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:47.200410 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1024/2062] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:46:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:46:47.516076 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d076607-73af-4bd4-bbca-99df1dd58d82 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:47.519149 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-5d076607-73af-4bd4-bbca-99df1dd58d82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:47.551633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d076607-73af-4bd4-bbca-99df1dd58d82 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:47.560895 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e710a2-b1e4-4b86-bb68-c8cfbcb4f7ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:47.561579 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e710a2-b1e4-4b86-bb68-c8cfbcb4f7ea admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:47.561859 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e710a2-b1e4-4b86-bb68-c8cfbcb4f7ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:47.562183 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e710a2-b1e4-4b86-bb68-c8cfbcb4f7ea admin admin] PATH_INFO: `/v3/projects/8461445392af4e50ba987c8a576e2dca` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:47.563334 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e710a2-b1e4-4b86-bb68-c8cfbcb4f7ea admin admin] RBAC: Authorizing `identity:delete_project(project_id=8461445392af4e50ba987c8a576e2dca)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:47.565236 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e710a2-b1e4-4b86-bb68-c8cfbcb4f7ea admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:47.611983 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:47.631833 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e9e710a2-b1e4-4b86-bb68-c8cfbcb4f7ea admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8461445392af4e50ba987c8a576e2dca', 'request_id': 'req-e9e710a2-b1e4-4b86-bb68-c8cfbcb4f7ea'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:47.638409 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e9e710a2-b1e4-4b86-bb68-c8cfbcb4f7ea admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8461445392af4e50ba987c8a576e2dca', 'request_id': 'req-e9e710a2-b1e4-4b86-bb68-c8cfbcb4f7ea'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:47.648874 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e9e710a2-b1e4-4b86-bb68-c8cfbcb4f7ea admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8461445392af4e50ba987c8a576e2dca', 'request_id': 'req-e9e710a2-b1e4-4b86-bb68-c8cfbcb4f7ea'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:47.660872 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e9e710a2-b1e4-4b86-bb68-c8cfbcb4f7ea admin admin] The token cache is being invalidate because project 8461445392af4e50ba987c8a576e2dca was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:46:47.661166 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e9e710a2-b1e4-4b86-bb68-c8cfbcb4f7ea admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:47.661859 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e9e710a2-b1e4-4b86-bb68-c8cfbcb4f7ea admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:47.663475 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1039/2063] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:46:47 2022] DELETE /identity/v3/projects/8461445392af4e50ba987c8a576e2dca => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:47.668029 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf75ffa-4d23-4123-9551-649a2085a912 service neutron] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:47.677884 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf75ffa-4d23-4123-9551-649a2085a912 service neutron] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:47.739072 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae54868-5ea0-4a79-9b97-01d45f336431 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:47.739650 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae54868-5ea0-4a79-9b97-01d45f336431 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:47.739930 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae54868-5ea0-4a79-9b97-01d45f336431 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:47.740204 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae54868-5ea0-4a79-9b97-01d45f336431 admin admin] PATH_INFO: `/v3/users/961e8d5d5e934299b1ff29b96f28cf8e` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:47.741355 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae54868-5ea0-4a79-9b97-01d45f336431 admin admin] RBAC: Authorizing `identity:delete_user(user_id=961e8d5d5e934299b1ff29b96f28cf8e)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:47.743390 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae54868-5ea0-4a79-9b97-01d45f336431 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:47.807773 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-2ae54868-5ea0-4a79-9b97-01d45f336431 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '961e8d5d5e934299b1ff29b96f28cf8e'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:47.816226 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-2ae54868-5ea0-4a79-9b97-01d45f336431 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '961e8d5d5e934299b1ff29b96f28cf8e'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:47.817353 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-2ae54868-5ea0-4a79-9b97-01d45f336431 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '961e8d5d5e934299b1ff29b96f28cf8e'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:47.817965 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-2ae54868-5ea0-4a79-9b97-01d45f336431 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '961e8d5d5e934299b1ff29b96f28cf8e'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:47.825957 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-2ae54868-5ea0-4a79-9b97-01d45f336431 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '961e8d5d5e934299b1ff29b96f28cf8e'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:47.847665 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1025/2064] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:46:47 2022] DELETE /identity/v3/users/961e8d5d5e934299b1ff29b96f28cf8e => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:46:48.391261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e710a2-b1e4-4b86-bb68-c8cfbcb4f7ea admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:48.393109 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e9e710a2-b1e4-4b86-bb68-c8cfbcb4f7ea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:48.407975 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e710a2-b1e4-4b86-bb68-c8cfbcb4f7ea admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:48.591159 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8e8f6f-cf7e-41c9-948c-cdf6a4cb10dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:48.591769 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8e8f6f-cf7e-41c9-948c-cdf6a4cb10dc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:48.592056 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8e8f6f-cf7e-41c9-948c-cdf6a4cb10dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:48.592332 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8e8f6f-cf7e-41c9-948c-cdf6a4cb10dc admin admin] PATH_INFO: `/v3/projects/2d878a488ebb45d8872c74df48985ff0` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:48.593494 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b8e8f6f-cf7e-41c9-948c-cdf6a4cb10dc admin admin] RBAC: Authorizing `identity:delete_project(project_id=2d878a488ebb45d8872c74df48985ff0)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:48.595462 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b8e8f6f-cf7e-41c9-948c-cdf6a4cb10dc admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:48.655466 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-1b8e8f6f-cf7e-41c9-948c-cdf6a4cb10dc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2d878a488ebb45d8872c74df48985ff0', 'request_id': 'req-1b8e8f6f-cf7e-41c9-948c-cdf6a4cb10dc'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:48.664177 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-1b8e8f6f-cf7e-41c9-948c-cdf6a4cb10dc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2d878a488ebb45d8872c74df48985ff0', 'request_id': 'req-1b8e8f6f-cf7e-41c9-948c-cdf6a4cb10dc'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:48.675395 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-1b8e8f6f-cf7e-41c9-948c-cdf6a4cb10dc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2d878a488ebb45d8872c74df48985ff0', 'request_id': 'req-1b8e8f6f-cf7e-41c9-948c-cdf6a4cb10dc'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:48.687125 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-1b8e8f6f-cf7e-41c9-948c-cdf6a4cb10dc admin admin] The token cache is being invalidate because project 2d878a488ebb45d8872c74df48985ff0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:46:48.687423 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-1b8e8f6f-cf7e-41c9-948c-cdf6a4cb10dc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:48.688181 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-1b8e8f6f-cf7e-41c9-948c-cdf6a4cb10dc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:48.690461 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1040/2065] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:46:48 2022] DELETE /identity/v3/projects/2d878a488ebb45d8872c74df48985ff0 => generated 0 bytes in 299 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:48.695235 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2607e99c-a2f3-4486-a1f9-8d4eb2e8f2a6 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:48.695957 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2607e99c-a2f3-4486-a1f9-8d4eb2e8f2a6 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:48.695957 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2607e99c-a2f3-4486-a1f9-8d4eb2e8f2a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:48.719377 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-2607e99c-a2f3-4486-a1f9-8d4eb2e8f2a6 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:48.720622 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-2607e99c-a2f3-4486-a1f9-8d4eb2e8f2a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:48.789285 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1026/2066] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:46:48.798058 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8da7201-921d-47d5-99ab-709f6121ad9f None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:48.798515 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8da7201-921d-47d5-99ab-709f6121ad9f None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:48.798893 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8da7201-921d-47d5-99ab-709f6121ad9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:48.819513 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-c8da7201-921d-47d5-99ab-709f6121ad9f None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:48.820734 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-c8da7201-921d-47d5-99ab-709f6121ad9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:48.828623 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1041/2067] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:46:48.833627 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2607e99c-a2f3-4486-a1f9-8d4eb2e8f2a6 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:48.843357 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2607e99c-a2f3-4486-a1f9-8d4eb2e8f2a6 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:48.849430 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7eb0399-0dcc-4725-acdf-081876efd9cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:48.850347 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7eb0399-0dcc-4725-acdf-081876efd9cb admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:48.850666 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7eb0399-0dcc-4725-acdf-081876efd9cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:48.850945 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7eb0399-0dcc-4725-acdf-081876efd9cb admin admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:48.851612 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7eb0399-0dcc-4725-acdf-081876efd9cb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:46:48.851918 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7eb0399-0dcc-4725-acdf-081876efd9cb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:48.853529 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7eb0399-0dcc-4725-acdf-081876efd9cb admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:48.863931 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:48.871994 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1027/2068] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:46:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:46:48.873380 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8da7201-921d-47d5-99ab-709f6121ad9f None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:48.884708 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8da7201-921d-47d5-99ab-709f6121ad9f None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:48.889447 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0344ef63-7263-4d00-977c-937d3c56b1e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:48.890052 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0344ef63-7263-4d00-977c-937d3c56b1e0 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:48.890334 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0344ef63-7263-4d00-977c-937d3c56b1e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:48.890634 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0344ef63-7263-4d00-977c-937d3c56b1e0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:48.891266 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0344ef63-7263-4d00-977c-937d3c56b1e0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:48.893265 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0344ef63-7263-4d00-977c-937d3c56b1e0 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:48.930788 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1042/2069] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:46:48 2022] POST /identity/v3/projects => generated 387 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:46:48.932124 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7eb0399-0dcc-4725-acdf-081876efd9cb admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:48.942338 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7eb0399-0dcc-4725-acdf-081876efd9cb admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:49.009315 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a2c323c-1a56-4fb1-9857-606c79c1df9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:49.010076 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2c323c-1a56-4fb1-9857-606c79c1df9d admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:49.010362 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2c323c-1a56-4fb1-9857-606c79c1df9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:49.010671 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2c323c-1a56-4fb1-9857-606c79c1df9d admin admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:49.011312 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a2c323c-1a56-4fb1-9857-606c79c1df9d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:49.013750 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a2c323c-1a56-4fb1-9857-606c79c1df9d admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:49.037067 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1028/2070] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:46:48 2022] POST /identity/v3/users => generated 294 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:46:49.041266 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0344ef63-7263-4d00-977c-937d3c56b1e0 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:49.052595 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0344ef63-7263-4d00-977c-937d3c56b1e0 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:49.057213 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1ccb31-9174-48c3-9686-175f9b4d6059 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:49.057809 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1ccb31-9174-48c3-9686-175f9b4d6059 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:49.058090 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1ccb31-9174-48c3-9686-175f9b4d6059 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:49.058372 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1ccb31-9174-48c3-9686-175f9b4d6059 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:49.059221 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac1ccb31-9174-48c3-9686-175f9b4d6059 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:49.061102 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac1ccb31-9174-48c3-9686-175f9b4d6059 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:49.072190 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1043/2071] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:46:49 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:46:49.076235 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a2c323c-1a56-4fb1-9857-606c79c1df9d admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:49.087018 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a2c323c-1a56-4fb1-9857-606c79c1df9d admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:49.092239 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16e54d7e-2a49-41e6-8707-53e95a274e4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:49.092869 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e54d7e-2a49-41e6-8707-53e95a274e4d admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:49.093152 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e54d7e-2a49-41e6-8707-53e95a274e4d admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:49.093512 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e54d7e-2a49-41e6-8707-53e95a274e4d admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:49.094230 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16e54d7e-2a49-41e6-8707-53e95a274e4d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:49.096033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16e54d7e-2a49-41e6-8707-53e95a274e4d admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:49.109001 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1029/2072] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:46:49 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:46:49.113248 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1ccb31-9174-48c3-9686-175f9b4d6059 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:49.123901 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1ccb31-9174-48c3-9686-175f9b4d6059 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:49.128802 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f9d83bd-6c54-4626-a6e2-e4fd5c687b75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:49.130075 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9d83bd-6c54-4626-a6e2-e4fd5c687b75 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:49.130367 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9d83bd-6c54-4626-a6e2-e4fd5c687b75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:49.130679 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9d83bd-6c54-4626-a6e2-e4fd5c687b75 admin admin] PATH_INFO: `/v3/projects/3255e6869df842b7acaa4e70f4c76dc7/users/3a7fc82c13ee4c07bbd1dc3d3a88056a/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:49.133441 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=85,keystone:DELETE=12,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:49.149854 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f9d83bd-6c54-4626-a6e2-e4fd5c687b75 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3255e6869df842b7acaa4e70f4c76dc7, user_id=3a7fc82c13ee4c07bbd1dc3d3a88056a, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:49.152571 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f9d83bd-6c54-4626-a6e2-e4fd5c687b75 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:49.176687 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1044/2073] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:46:49 2022] PUT /identity/v3/projects/3255e6869df842b7acaa4e70f4c76dc7/users/3a7fc82c13ee4c07bbd1dc3d3a88056a/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:49.182083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7826a728-7f3f-4fe3-b2da-a56707d1aeb9 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:49.182370 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7826a728-7f3f-4fe3-b2da-a56707d1aeb9 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:49.182673 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7826a728-7f3f-4fe3-b2da-a56707d1aeb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:49.214859 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-7826a728-7f3f-4fe3-b2da-a56707d1aeb9 None None] MFA Rules not processed for user `3a7fc82c13ee4c07bbd1dc3d3a88056a`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:49.216987 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-7826a728-7f3f-4fe3-b2da-a56707d1aeb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:49.315333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1030/2074] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:49 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:46:49.320328 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f9d83bd-6c54-4626-a6e2-e4fd5c687b75 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:49.398751 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f9d83bd-6c54-4626-a6e2-e4fd5c687b75 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:49.407554 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d63d636d-d82d-4585-92bc-998805a5134f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:49.408828 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63d636d-d82d-4585-92bc-998805a5134f admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:49.409284 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63d636d-d82d-4585-92bc-998805a5134f admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:49.409741 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63d636d-d82d-4585-92bc-998805a5134f admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:49.410947 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d63d636d-d82d-4585-92bc-998805a5134f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:49.413106 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d63d636d-d82d-4585-92bc-998805a5134f admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:49.445793 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1045/2075] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:46:49 2022] POST /identity/v3/projects => generated 387 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:46:49.449852 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7826a728-7f3f-4fe3-b2da-a56707d1aeb9 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:49.462488 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7826a728-7f3f-4fe3-b2da-a56707d1aeb9 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:49.543043 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd94d495-a788-4fb0-811d-6f559f6acdd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:49.543671 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd94d495-a788-4fb0-811d-6f559f6acdd7 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:49.544167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd94d495-a788-4fb0-811d-6f559f6acdd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:49.544456 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd94d495-a788-4fb0-811d-6f559f6acdd7 admin admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:49.545225 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd94d495-a788-4fb0-811d-6f559f6acdd7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:49.547274 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd94d495-a788-4fb0-811d-6f559f6acdd7 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:49.570107 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:49.583171 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1031/2076] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:46:49 2022] POST /identity/v3/users => generated 300 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:46:49.584098 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d63d636d-d82d-4585-92bc-998805a5134f admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:49.605610 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d63d636d-d82d-4585-92bc-998805a5134f admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:49.615020 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b738a9-3734-4784-8b8f-762a8f60ff4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:49.616267 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b738a9-3734-4784-8b8f-762a8f60ff4c admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:49.617111 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b738a9-3734-4784-8b8f-762a8f60ff4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:49.617579 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b738a9-3734-4784-8b8f-762a8f60ff4c admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:49.618250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21b738a9-3734-4784-8b8f-762a8f60ff4c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:49.624116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21b738a9-3734-4784-8b8f-762a8f60ff4c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:49.639076 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1046/2077] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:46:49 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:46:49.643000 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd94d495-a788-4fb0-811d-6f559f6acdd7 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:49.661311 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd94d495-a788-4fb0-811d-6f559f6acdd7 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:49.668225 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a26b39a5-e5ae-4353-8c39-8f9cc1b518ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:49.669058 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26b39a5-e5ae-4353-8c39-8f9cc1b518ed admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:49.669355 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26b39a5-e5ae-4353-8c39-8f9cc1b518ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:49.669637 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26b39a5-e5ae-4353-8c39-8f9cc1b518ed admin admin] PATH_INFO: `/v3/projects/f3ae38c153134065b04a3eac92064bd2/users/466b4a75b61e4e67b0950f63c8da00bb/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:49.688559 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a26b39a5-e5ae-4353-8c39-8f9cc1b518ed admin admin] RBAC: Authorizing `identity:create_grant(project_id=f3ae38c153134065b04a3eac92064bd2, user_id=466b4a75b61e4e67b0950f63c8da00bb, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:49.691120 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a26b39a5-e5ae-4353-8c39-8f9cc1b518ed admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:49.735885 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1032/2078] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:46:49 2022] PUT /identity/v3/projects/f3ae38c153134065b04a3eac92064bd2/users/466b4a75b61e4e67b0950f63c8da00bb/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 93 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:49.740523 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b738a9-3734-4784-8b8f-762a8f60ff4c admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:49.751050 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b738a9-3734-4784-8b8f-762a8f60ff4c admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:49.854216 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-855c4053-b361-4228-aebe-e10da2071051 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:49.855101 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855c4053-b361-4228-aebe-e10da2071051 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:49.855394 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855c4053-b361-4228-aebe-e10da2071051 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:49.855744 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855c4053-b361-4228-aebe-e10da2071051 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:49.856426 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-855c4053-b361-4228-aebe-e10da2071051 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:49.858058 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-855c4053-b361-4228-aebe-e10da2071051 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:49.869219 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1047/2079] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:46:49 2022] GET /identity/v3/roles => generated 1420 bytes in 129 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:46:49.873423 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a26b39a5-e5ae-4353-8c39-8f9cc1b518ed admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:49.884834 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a26b39a5-e5ae-4353-8c39-8f9cc1b518ed admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:49.889969 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af490518-4ab6-4ad5-a435-4a29531da2f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:49.890274 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af490518-4ab6-4ad5-a435-4a29531da2f1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:49.890598 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af490518-4ab6-4ad5-a435-4a29531da2f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:49.890875 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af490518-4ab6-4ad5-a435-4a29531da2f1 admin admin] PATH_INFO: `/v3/projects/f3ae38c153134065b04a3eac92064bd2/users/466b4a75b61e4e67b0950f63c8da00bb/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:49.892791 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af490518-4ab6-4ad5-a435-4a29531da2f1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f3ae38c153134065b04a3eac92064bd2, user_id=466b4a75b61e4e67b0950f63c8da00bb, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:49.896093 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af490518-4ab6-4ad5-a435-4a29531da2f1 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:49.908778 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1033/2080] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:46:49 2022] PUT /identity/v3/projects/f3ae38c153134065b04a3eac92064bd2/users/466b4a75b61e4e67b0950f63c8da00bb/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:49.919754 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8daa8b0-dfe5-4c84-ba45-39189f4e7d58 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:49.920108 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8daa8b0-dfe5-4c84-ba45-39189f4e7d58 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:49.920496 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8daa8b0-dfe5-4c84-ba45-39189f4e7d58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:49.954717 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-b8daa8b0-dfe5-4c84-ba45-39189f4e7d58 None None] MFA Rules not processed for user `466b4a75b61e4e67b0950f63c8da00bb`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:49.956020 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-b8daa8b0-dfe5-4c84-ba45-39189f4e7d58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:50.002010 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:50.034612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1048/2081] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:49 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:46:50.046711 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af490518-4ab6-4ad5-a435-4a29531da2f1 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:50.048462 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-af490518-4ab6-4ad5-a435-4a29531da2f1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:50.110219 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af490518-4ab6-4ad5-a435-4a29531da2f1 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:50.114596 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d00f5af-38c0-468c-a983-d4274b233872 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:50.115219 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d00f5af-38c0-468c-a983-d4274b233872 service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:50.115502 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d00f5af-38c0-468c-a983-d4274b233872 service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:50.115778 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d00f5af-38c0-468c-a983-d4274b233872 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:50.125556 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d00f5af-38c0-468c-a983-d4274b233872 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:50.127055 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d00f5af-38c0-468c-a983-d4274b233872 service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:50.137331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1034/2082] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:46:50 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:46:50.288635 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8daa8b0-dfe5-4c84-ba45-39189f4e7d58 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:50.365231 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8daa8b0-dfe5-4c84-ba45-39189f4e7d58 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:50.374361 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb31db0-5d38-4b96-a53a-17de98d859b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:50.375048 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb31db0-5d38-4b96-a53a-17de98d859b2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:50.375331 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb31db0-5d38-4b96-a53a-17de98d859b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:50.379520 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb31db0-5d38-4b96-a53a-17de98d859b2 admin admin] PATH_INFO: `/v3/users/3a7fc82c13ee4c07bbd1dc3d3a88056a` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:50.381004 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb31db0-5d38-4b96-a53a-17de98d859b2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3a7fc82c13ee4c07bbd1dc3d3a88056a)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:50.383041 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb31db0-5d38-4b96-a53a-17de98d859b2 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:50.445808 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ebb31db0-5d38-4b96-a53a-17de98d859b2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3a7fc82c13ee4c07bbd1dc3d3a88056a'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:50.455560 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ebb31db0-5d38-4b96-a53a-17de98d859b2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3a7fc82c13ee4c07bbd1dc3d3a88056a'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:50.456356 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ebb31db0-5d38-4b96-a53a-17de98d859b2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3a7fc82c13ee4c07bbd1dc3d3a88056a'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:50.457088 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ebb31db0-5d38-4b96-a53a-17de98d859b2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3a7fc82c13ee4c07bbd1dc3d3a88056a'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:50.466135 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ebb31db0-5d38-4b96-a53a-17de98d859b2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3a7fc82c13ee4c07bbd1dc3d3a88056a'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:50.494706 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1049/2083] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:46:50 2022] DELETE /identity/v3/users/3a7fc82c13ee4c07bbd1dc3d3a88056a => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:50.499450 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3187593f-8f97-4bef-841e-2ca9568e8580 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:50.500052 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3187593f-8f97-4bef-841e-2ca9568e8580 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:50.500547 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3187593f-8f97-4bef-841e-2ca9568e8580 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:50.523926 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-3187593f-8f97-4bef-841e-2ca9568e8580 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:50.525247 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-3187593f-8f97-4bef-841e-2ca9568e8580 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:50.606234 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1035/2084] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:46:50.623243 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb31db0-5d38-4b96-a53a-17de98d859b2 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:50.624687 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ebb31db0-5d38-4b96-a53a-17de98d859b2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:50.688423 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb31db0-5d38-4b96-a53a-17de98d859b2 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:50.692670 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9e9f42-9437-4267-87c5-c4a07249c723 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:50.693270 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9e9f42-9437-4267-87c5-c4a07249c723 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:50.693652 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9e9f42-9437-4267-87c5-c4a07249c723 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:50.694074 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9e9f42-9437-4267-87c5-c4a07249c723 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:50.710239 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f9e9f42-9437-4267-87c5-c4a07249c723 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:50.711632 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f9e9f42-9437-4267-87c5-c4a07249c723 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:50.717766 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:50.731913 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1050/2085] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:46:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:46:51.140815 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3187593f-8f97-4bef-841e-2ca9568e8580 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:51.142579 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-3187593f-8f97-4bef-841e-2ca9568e8580 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:51.200703 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3187593f-8f97-4bef-841e-2ca9568e8580 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:51.217262 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b3c4347-ec73-48bf-a59b-a924365a4df7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:51.218118 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3c4347-ec73-48bf-a59b-a924365a4df7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:51.218574 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3c4347-ec73-48bf-a59b-a924365a4df7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:51.219066 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3c4347-ec73-48bf-a59b-a924365a4df7 admin admin] PATH_INFO: `/v3/projects/3255e6869df842b7acaa4e70f4c76dc7` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:51.220424 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b3c4347-ec73-48bf-a59b-a924365a4df7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3255e6869df842b7acaa4e70f4c76dc7)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:51.223253 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b3c4347-ec73-48bf-a59b-a924365a4df7 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:51.293640 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-0b3c4347-ec73-48bf-a59b-a924365a4df7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3255e6869df842b7acaa4e70f4c76dc7', 'request_id': 'req-0b3c4347-ec73-48bf-a59b-a924365a4df7'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:51.300532 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-0b3c4347-ec73-48bf-a59b-a924365a4df7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3255e6869df842b7acaa4e70f4c76dc7', 'request_id': 'req-0b3c4347-ec73-48bf-a59b-a924365a4df7'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:51.306069 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=92,keystone:INSERT=3,keystone:DELETE=5 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:51.309553 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-0b3c4347-ec73-48bf-a59b-a924365a4df7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3255e6869df842b7acaa4e70f4c76dc7', 'request_id': 'req-0b3c4347-ec73-48bf-a59b-a924365a4df7'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:51.327387 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-0b3c4347-ec73-48bf-a59b-a924365a4df7 admin admin] The token cache is being invalidate because project 3255e6869df842b7acaa4e70f4c76dc7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:46:51.328296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-0b3c4347-ec73-48bf-a59b-a924365a4df7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:51.329037 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-0b3c4347-ec73-48bf-a59b-a924365a4df7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:51.333093 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1036/2086] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:46:51 2022] DELETE /identity/v3/projects/3255e6869df842b7acaa4e70f4c76dc7 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:51.335029 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9e9f42-9437-4267-87c5-c4a07249c723 service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:51.345155 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9e9f42-9437-4267-87c5-c4a07249c723 service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:51.439355 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a22ba063-488c-4b65-87af-f2fabe71b3ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:51.440301 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22ba063-488c-4b65-87af-f2fabe71b3ed admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:51.440690 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22ba063-488c-4b65-87af-f2fabe71b3ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:51.441084 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22ba063-488c-4b65-87af-f2fabe71b3ed admin admin] PATH_INFO: `/v3/users/466b4a75b61e4e67b0950f63c8da00bb` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:51.442933 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a22ba063-488c-4b65-87af-f2fabe71b3ed admin admin] RBAC: Authorizing `identity:delete_user(user_id=466b4a75b61e4e67b0950f63c8da00bb)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:51.456432 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a22ba063-488c-4b65-87af-f2fabe71b3ed admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:51.532251 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a22ba063-488c-4b65-87af-f2fabe71b3ed admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '466b4a75b61e4e67b0950f63c8da00bb'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:51.542821 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a22ba063-488c-4b65-87af-f2fabe71b3ed admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '466b4a75b61e4e67b0950f63c8da00bb'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:51.545050 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a22ba063-488c-4b65-87af-f2fabe71b3ed admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '466b4a75b61e4e67b0950f63c8da00bb'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:51.545994 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a22ba063-488c-4b65-87af-f2fabe71b3ed admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '466b4a75b61e4e67b0950f63c8da00bb'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:51.561867 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a22ba063-488c-4b65-87af-f2fabe71b3ed admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '466b4a75b61e4e67b0950f63c8da00bb'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:51.600768 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1051/2087] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:46:51 2022] DELETE /identity/v3/users/466b4a75b61e4e67b0950f63c8da00bb => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:46:51.967094 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b3c4347-ec73-48bf-a59b-a924365a4df7 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:51.968534 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-0b3c4347-ec73-48bf-a59b-a924365a4df7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:51.986743 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b3c4347-ec73-48bf-a59b-a924365a4df7 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:52.066678 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6dec88-6b40-4227-9714-1cf821519126 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:52.067422 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6dec88-6b40-4227-9714-1cf821519126 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:52.067771 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6dec88-6b40-4227-9714-1cf821519126 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:52.068115 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6dec88-6b40-4227-9714-1cf821519126 admin admin] PATH_INFO: `/v3/projects/f3ae38c153134065b04a3eac92064bd2` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:52.069516 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d6dec88-6b40-4227-9714-1cf821519126 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f3ae38c153134065b04a3eac92064bd2)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:52.071564 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d6dec88-6b40-4227-9714-1cf821519126 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:52.122813 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-0d6dec88-6b40-4227-9714-1cf821519126 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f3ae38c153134065b04a3eac92064bd2', 'request_id': 'req-0d6dec88-6b40-4227-9714-1cf821519126'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:52.129512 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-0d6dec88-6b40-4227-9714-1cf821519126 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f3ae38c153134065b04a3eac92064bd2', 'request_id': 'req-0d6dec88-6b40-4227-9714-1cf821519126'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:52.144489 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-0d6dec88-6b40-4227-9714-1cf821519126 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f3ae38c153134065b04a3eac92064bd2', 'request_id': 'req-0d6dec88-6b40-4227-9714-1cf821519126'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:52.162422 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-0d6dec88-6b40-4227-9714-1cf821519126 admin admin] The token cache is being invalidate because project f3ae38c153134065b04a3eac92064bd2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:46:52.162794 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-0d6dec88-6b40-4227-9714-1cf821519126 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:52.163761 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-0d6dec88-6b40-4227-9714-1cf821519126 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:46:52.169136 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1037/2088] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:46:51 2022] DELETE /identity/v3/projects/f3ae38c153134065b04a3eac92064bd2 => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:52.170902 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32b2c08-1245-4bd0-8b72-baf390c01112 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:52.171933 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32b2c08-1245-4bd0-8b72-baf390c01112 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:52.172284 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32b2c08-1245-4bd0-8b72-baf390c01112 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:52.213839 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-e32b2c08-1245-4bd0-8b72-baf390c01112 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:52.215205 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e32b2c08-1245-4bd0-8b72-baf390c01112 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:52.364758 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1052/2089] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:46:52.372783 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e2302f-1ec7-4bf5-9ff3-3b5a374c4506 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:52.373275 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e2302f-1ec7-4bf5-9ff3-3b5a374c4506 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:52.373638 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e2302f-1ec7-4bf5-9ff3-3b5a374c4506 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:52.397242 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-d7e2302f-1ec7-4bf5-9ff3-3b5a374c4506 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:52.398960 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-d7e2302f-1ec7-4bf5-9ff3-3b5a374c4506 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:52.408964 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1038/2090] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:46:52.413226 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e32b2c08-1245-4bd0-8b72-baf390c01112 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:52.423883 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e32b2c08-1245-4bd0-8b72-baf390c01112 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:52.431075 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8566dcb-064c-479b-a52c-dcc8357c45ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:52.431885 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8566dcb-064c-479b-a52c-dcc8357c45ac admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:52.432244 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8566dcb-064c-479b-a52c-dcc8357c45ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:52.432509 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8566dcb-064c-479b-a52c-dcc8357c45ac admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:52.433267 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8566dcb-064c-479b-a52c-dcc8357c45ac admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:46:52.433748 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8566dcb-064c-479b-a52c-dcc8357c45ac admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:52.435619 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8566dcb-064c-479b-a52c-dcc8357c45ac admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:52.464205 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1053/2091] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:46:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:46:52.468767 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e2302f-1ec7-4bf5-9ff3-3b5a374c4506 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:52.478556 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e2302f-1ec7-4bf5-9ff3-3b5a374c4506 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:52.483258 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcdc80de-e768-4695-bbc2-b0a7406d9e1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:52.483869 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcdc80de-e768-4695-bbc2-b0a7406d9e1d admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:52.484154 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcdc80de-e768-4695-bbc2-b0a7406d9e1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:52.484430 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcdc80de-e768-4695-bbc2-b0a7406d9e1d admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:52.485082 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcdc80de-e768-4695-bbc2-b0a7406d9e1d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:52.487082 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcdc80de-e768-4695-bbc2-b0a7406d9e1d admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:52.524337 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1039/2092] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:46:52 2022] POST /identity/v3/projects => generated 375 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:46:52.528363 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8566dcb-064c-479b-a52c-dcc8357c45ac admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:52.544862 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8566dcb-064c-479b-a52c-dcc8357c45ac admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:52.557307 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:52.658767 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4feb1057-5c43-49f8-b1e9-116cfdfeed1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:52.660367 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4feb1057-5c43-49f8-b1e9-116cfdfeed1b admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:52.662420 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4feb1057-5c43-49f8-b1e9-116cfdfeed1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:52.662845 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4feb1057-5c43-49f8-b1e9-116cfdfeed1b admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:52.665128 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4feb1057-5c43-49f8-b1e9-116cfdfeed1b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:52.670553 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4feb1057-5c43-49f8-b1e9-116cfdfeed1b admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:52.702590 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1054/2093] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:46:52 2022] POST /identity/v3/users => generated 288 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:46:52.705472 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcdc80de-e768-4695-bbc2-b0a7406d9e1d admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:52.717244 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcdc80de-e768-4695-bbc2-b0a7406d9e1d admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:52.722135 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff85f8ea-7f3e-4c6d-a8be-4fcf2450b36a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:52.722829 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff85f8ea-7f3e-4c6d-a8be-4fcf2450b36a admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:52.723183 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff85f8ea-7f3e-4c6d-a8be-4fcf2450b36a admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:52.723502 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff85f8ea-7f3e-4c6d-a8be-4fcf2450b36a admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:52.724170 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff85f8ea-7f3e-4c6d-a8be-4fcf2450b36a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:52.725821 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff85f8ea-7f3e-4c6d-a8be-4fcf2450b36a admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:52.739217 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1040/2094] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:46:52 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:46:52.742729 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4feb1057-5c43-49f8-b1e9-116cfdfeed1b admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:52.753881 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4feb1057-5c43-49f8-b1e9-116cfdfeed1b admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:52.759379 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121418ce-573c-4afd-ba0d-241e184685e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:52.760026 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121418ce-573c-4afd-ba0d-241e184685e5 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:52.760306 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121418ce-573c-4afd-ba0d-241e184685e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:52.760600 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121418ce-573c-4afd-ba0d-241e184685e5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:52.761233 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-121418ce-573c-4afd-ba0d-241e184685e5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:52.762864 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-121418ce-573c-4afd-ba0d-241e184685e5 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:52.774842 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1055/2095] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:46:52 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:46:52.780524 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff85f8ea-7f3e-4c6d-a8be-4fcf2450b36a admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:52.791719 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff85f8ea-7f3e-4c6d-a8be-4fcf2450b36a admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:52.796597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4155c6-0ae1-40a9-a230-291ad9169212 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:52.797290 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4155c6-0ae1-40a9-a230-291ad9169212 admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:52.797572 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4155c6-0ae1-40a9-a230-291ad9169212 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:52.797847 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d4155c6-0ae1-40a9-a230-291ad9169212 admin admin] PATH_INFO: `/v3/projects/aa635ba0a62a4806af2a9fac99e19a02/users/9a9a75d645ce406ba22561e2f2b7b2cf/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:52.813660 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d4155c6-0ae1-40a9-a230-291ad9169212 admin admin] RBAC: Authorizing `identity:create_grant(project_id=aa635ba0a62a4806af2a9fac99e19a02, user_id=9a9a75d645ce406ba22561e2f2b7b2cf, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:52.816470 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d4155c6-0ae1-40a9-a230-291ad9169212 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:52.831945 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1041/2096] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:46:52 2022] PUT /identity/v3/projects/aa635ba0a62a4806af2a9fac99e19a02/users/9a9a75d645ce406ba22561e2f2b7b2cf/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:46:52.843922 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db41d78-e899-43f8-9974-0644c728ff98 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:52.844323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db41d78-e899-43f8-9974-0644c728ff98 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:52.844644 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db41d78-e899-43f8-9974-0644c728ff98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:52.878811 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-1db41d78-e899-43f8-9974-0644c728ff98 None None] MFA Rules not processed for user `9a9a75d645ce406ba22561e2f2b7b2cf`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:46:52.880781 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-1db41d78-e899-43f8-9974-0644c728ff98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:52.966461 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1056/2097] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:46:52 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:46:52.988666 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4155c6-0ae1-40a9-a230-291ad9169212 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:46:52.990535 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-7d4155c6-0ae1-40a9-a230-291ad9169212 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:46:53.010608 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=88,keystone:DELETE=9,keystone:INSERT=3 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:46:53.071541 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d4155c6-0ae1-40a9-a230-291ad9169212 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:46:53.075505 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495b702c-7b96-4373-9d49-11616bbe26ae service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:46:53.076198 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495b702c-7b96-4373-9d49-11616bbe26ae service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:46:53.076479 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495b702c-7b96-4373-9d49-11616bbe26ae service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:46:53.076754 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495b702c-7b96-4373-9d49-11616bbe26ae service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:46:53.086909 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-495b702c-7b96-4373-9d49-11616bbe26ae service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:46:53.088378 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-495b702c-7b96-4373-9d49-11616bbe26ae service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:46:53.097725 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1042/2098] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:46:52 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:47:02.962133 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=63,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:47:03.093479 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=20 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:47:17.251979 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db41d78-e899-43f8-9974-0644c728ff98 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:17.253461 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-1db41d78-e899-43f8-9974-0644c728ff98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:17.323817 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db41d78-e899-43f8-9974-0644c728ff98 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:17.330838 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6c8728-732c-449d-b52c-6126e130f7cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:17.331551 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6c8728-732c-449d-b52c-6126e130f7cc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:17.332581 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6c8728-732c-449d-b52c-6126e130f7cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:17.332718 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6c8728-732c-449d-b52c-6126e130f7cc admin admin] PATH_INFO: `/v3/users/802938c8b66c4ee59d8ba9c3bdc5c534` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:17.333899 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d6c8728-732c-449d-b52c-6126e130f7cc admin admin] RBAC: Authorizing `identity:delete_user(user_id=802938c8b66c4ee59d8ba9c3bdc5c534)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:17.335917 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d6c8728-732c-449d-b52c-6126e130f7cc admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:17.392123 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-4d6c8728-732c-449d-b52c-6126e130f7cc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '802938c8b66c4ee59d8ba9c3bdc5c534'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:17.400042 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-4d6c8728-732c-449d-b52c-6126e130f7cc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '802938c8b66c4ee59d8ba9c3bdc5c534'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:17.400953 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-4d6c8728-732c-449d-b52c-6126e130f7cc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '802938c8b66c4ee59d8ba9c3bdc5c534'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:17.401615 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-4d6c8728-732c-449d-b52c-6126e130f7cc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '802938c8b66c4ee59d8ba9c3bdc5c534'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:17.409127 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-4d6c8728-732c-449d-b52c-6126e130f7cc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '802938c8b66c4ee59d8ba9c3bdc5c534'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:17.428083 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1057/2099] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:47:17 2022] DELETE /identity/v3/users/802938c8b66c4ee59d8ba9c3bdc5c534 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:47:17.432592 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7f2a65-9800-4802-a05e-5f4f922c3a89 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:17.433065 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7f2a65-9800-4802-a05e-5f4f922c3a89 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:17.433218 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7f2a65-9800-4802-a05e-5f4f922c3a89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:17.452795 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-0c7f2a65-9800-4802-a05e-5f4f922c3a89 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:47:17.454257 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-0c7f2a65-9800-4802-a05e-5f4f922c3a89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:17.525570 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1043/2100] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:47:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:47:17.532680 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6c8728-732c-449d-b52c-6126e130f7cc admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:17.534128 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4d6c8728-732c-449d-b52c-6126e130f7cc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:17.604111 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6c8728-732c-449d-b52c-6126e130f7cc admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:17.608030 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822b3a26-b29f-416b-8ce4-f66d252d1474 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:17.609107 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822b3a26-b29f-416b-8ce4-f66d252d1474 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:17.609354 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822b3a26-b29f-416b-8ce4-f66d252d1474 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:17.609628 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822b3a26-b29f-416b-8ce4-f66d252d1474 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:17.619711 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-822b3a26-b29f-416b-8ce4-f66d252d1474 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:17.621145 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-822b3a26-b29f-416b-8ce4-f66d252d1474 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:17.631929 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1058/2101] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:47:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:47:17.942528 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c7f2a65-9800-4802-a05e-5f4f922c3a89 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:17.944075 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-0c7f2a65-9800-4802-a05e-5f4f922c3a89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:17.956610 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c7f2a65-9800-4802-a05e-5f4f922c3a89 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:17.963203 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb01adfe-0f50-47d9-9788-2c79fbbb5251 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:17.964617 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb01adfe-0f50-47d9-9788-2c79fbbb5251 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:17.965549 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb01adfe-0f50-47d9-9788-2c79fbbb5251 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:17.966310 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb01adfe-0f50-47d9-9788-2c79fbbb5251 admin admin] PATH_INFO: `/v3/projects/e755446b01eb49f081e9ce5b2442b503` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:17.968197 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb01adfe-0f50-47d9-9788-2c79fbbb5251 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e755446b01eb49f081e9ce5b2442b503)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:17.970769 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb01adfe-0f50-47d9-9788-2c79fbbb5251 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:18.047178 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-fb01adfe-0f50-47d9-9788-2c79fbbb5251 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e755446b01eb49f081e9ce5b2442b503', 'request_id': 'req-fb01adfe-0f50-47d9-9788-2c79fbbb5251'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:18.054039 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-fb01adfe-0f50-47d9-9788-2c79fbbb5251 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e755446b01eb49f081e9ce5b2442b503', 'request_id': 'req-fb01adfe-0f50-47d9-9788-2c79fbbb5251'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:18.073375 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-fb01adfe-0f50-47d9-9788-2c79fbbb5251 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e755446b01eb49f081e9ce5b2442b503', 'request_id': 'req-fb01adfe-0f50-47d9-9788-2c79fbbb5251'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:18.087409 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-fb01adfe-0f50-47d9-9788-2c79fbbb5251 admin admin] The token cache is being invalidate because project e755446b01eb49f081e9ce5b2442b503 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:47:18.087723 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-fb01adfe-0f50-47d9-9788-2c79fbbb5251 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:18.088443 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-fb01adfe-0f50-47d9-9788-2c79fbbb5251 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:18.089935 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1044/2102] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:47:17 2022] DELETE /identity/v3/projects/e755446b01eb49f081e9ce5b2442b503 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:47:18.094892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a2ebcc-dbd6-4222-913a-3585c6a796bf None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:18.095215 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a2ebcc-dbd6-4222-913a-3585c6a796bf None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:18.095509 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a2ebcc-dbd6-4222-913a-3585c6a796bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:18.115759 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-d4a2ebcc-dbd6-4222-913a-3585c6a796bf None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:47:18.117145 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d4a2ebcc-dbd6-4222-913a-3585c6a796bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:18.134162 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:47:18.180411 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1059/2103] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:47:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:47:18.188813 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b3638f-e658-471d-89ea-7f33cfff5637 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:18.189304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b3638f-e658-471d-89ea-7f33cfff5637 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:18.189705 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b3638f-e658-471d-89ea-7f33cfff5637 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:18.211475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-37b3638f-e658-471d-89ea-7f33cfff5637 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:47:18.212743 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-37b3638f-e658-471d-89ea-7f33cfff5637 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:18.220849 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1045/2104] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:47:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:47:18.225655 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a2ebcc-dbd6-4222-913a-3585c6a796bf None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:18.235624 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a2ebcc-dbd6-4222-913a-3585c6a796bf None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:18.240283 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e11e0f1-03cf-4696-8789-42bfe00125ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:18.240948 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e11e0f1-03cf-4696-8789-42bfe00125ed admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:18.241229 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e11e0f1-03cf-4696-8789-42bfe00125ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:18.241503 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e11e0f1-03cf-4696-8789-42bfe00125ed admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:18.242145 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e11e0f1-03cf-4696-8789-42bfe00125ed admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:47:18.242485 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e11e0f1-03cf-4696-8789-42bfe00125ed admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:18.244056 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e11e0f1-03cf-4696-8789-42bfe00125ed admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:18.256625 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1060/2105] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:47:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:47:18.260546 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b3638f-e658-471d-89ea-7f33cfff5637 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:18.270079 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b3638f-e658-471d-89ea-7f33cfff5637 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:18.274773 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-774962de-6cb1-47e9-bfdf-76a2e4bb138f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:18.275366 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774962de-6cb1-47e9-bfdf-76a2e4bb138f admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:18.275642 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774962de-6cb1-47e9-bfdf-76a2e4bb138f admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:18.275909 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774962de-6cb1-47e9-bfdf-76a2e4bb138f admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:18.276561 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-774962de-6cb1-47e9-bfdf-76a2e4bb138f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:18.278494 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-774962de-6cb1-47e9-bfdf-76a2e4bb138f admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:18.308738 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1046/2106] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:47:18 2022] POST /identity/v3/projects => generated 383 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:47:18.322087 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e11e0f1-03cf-4696-8789-42bfe00125ed admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:18.324304 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e11e0f1-03cf-4696-8789-42bfe00125ed admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:18.388293 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8195079c-229e-4303-95ec-cd6241491d34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:18.388883 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8195079c-229e-4303-95ec-cd6241491d34 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:18.389391 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8195079c-229e-4303-95ec-cd6241491d34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:18.389720 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8195079c-229e-4303-95ec-cd6241491d34 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:18.390379 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8195079c-229e-4303-95ec-cd6241491d34 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:18.392363 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8195079c-229e-4303-95ec-cd6241491d34 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:18.420697 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1061/2107] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:47:18 2022] POST /identity/v3/users => generated 292 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:47:18.421917 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-774962de-6cb1-47e9-bfdf-76a2e4bb138f admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:18.433619 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-774962de-6cb1-47e9-bfdf-76a2e4bb138f admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:18.438640 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec5cfc7-bd02-4dff-9c46-6dca5c9e0ced admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:18.439385 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec5cfc7-bd02-4dff-9c46-6dca5c9e0ced admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:18.439760 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec5cfc7-bd02-4dff-9c46-6dca5c9e0ced admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:18.440131 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec5cfc7-bd02-4dff-9c46-6dca5c9e0ced admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:18.441853 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec5cfc7-bd02-4dff-9c46-6dca5c9e0ced admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:18.443596 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cec5cfc7-bd02-4dff-9c46-6dca5c9e0ced admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:18.455142 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1047/2108] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:47:18 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:47:18.459374 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8195079c-229e-4303-95ec-cd6241491d34 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:18.469716 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8195079c-229e-4303-95ec-cd6241491d34 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:18.476802 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc3a6d8-f800-4427-b892-88e1a7f7741f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:18.477476 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc3a6d8-f800-4427-b892-88e1a7f7741f admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:18.477695 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc3a6d8-f800-4427-b892-88e1a7f7741f admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:18.478225 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc3a6d8-f800-4427-b892-88e1a7f7741f admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:18.478799 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dc3a6d8-f800-4427-b892-88e1a7f7741f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:18.480400 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dc3a6d8-f800-4427-b892-88e1a7f7741f admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:18.492024 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1062/2109] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:47:18 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:47:18.496067 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec5cfc7-bd02-4dff-9c46-6dca5c9e0ced admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:18.506970 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec5cfc7-bd02-4dff-9c46-6dca5c9e0ced admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:18.511558 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5559a79a-7ff4-4642-8091-18943c1bc3e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:18.512335 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5559a79a-7ff4-4642-8091-18943c1bc3e4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:18.512669 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5559a79a-7ff4-4642-8091-18943c1bc3e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:18.513007 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5559a79a-7ff4-4642-8091-18943c1bc3e4 admin admin] PATH_INFO: `/v3/projects/9a20b51b10ce463da8f9ad280063bb09/users/ec23c6593e7a446298d607294997664c/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:18.528538 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5559a79a-7ff4-4642-8091-18943c1bc3e4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9a20b51b10ce463da8f9ad280063bb09, user_id=ec23c6593e7a446298d607294997664c, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:18.532838 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5559a79a-7ff4-4642-8091-18943c1bc3e4 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:18.559589 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1048/2110] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:47:18 2022] PUT /identity/v3/projects/9a20b51b10ce463da8f9ad280063bb09/users/ec23c6593e7a446298d607294997664c/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:47:18.564624 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80619255-8e99-4b20-a776-36f5982d9aee None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:18.565024 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80619255-8e99-4b20-a776-36f5982d9aee None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:18.565373 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80619255-8e99-4b20-a776-36f5982d9aee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:18.592651 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-80619255-8e99-4b20-a776-36f5982d9aee None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:47:18.593905 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-80619255-8e99-4b20-a776-36f5982d9aee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:18.689911 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1063/2111] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:47:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:47:18.700742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5559a79a-7ff4-4642-8091-18943c1bc3e4 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:18.703690 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-5559a79a-7ff4-4642-8091-18943c1bc3e4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:18.736514 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:47:18.784115 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5559a79a-7ff4-4642-8091-18943c1bc3e4 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:18.787936 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0ab6f4-f24f-4ee0-ae05-1384f4a0a813 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:18.788555 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0ab6f4-f24f-4ee0-ae05-1384f4a0a813 service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:18.788923 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0ab6f4-f24f-4ee0-ae05-1384f4a0a813 service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:18.789121 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0ab6f4-f24f-4ee0-ae05-1384f4a0a813 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:18.799104 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0ab6f4-f24f-4ee0-ae05-1384f4a0a813 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:18.800564 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0ab6f4-f24f-4ee0-ae05-1384f4a0a813 service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:18.810777 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1049/2112] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:47:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:47:19.523712 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34bed32-0175-4cd1-a742-7d0805d2349a None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:19.524662 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34bed32-0175-4cd1-a742-7d0805d2349a None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:19.524949 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34bed32-0175-4cd1-a742-7d0805d2349a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:19.561315 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-d34bed32-0175-4cd1-a742-7d0805d2349a None None] MFA Rules not processed for user `ec23c6593e7a446298d607294997664c`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:47:19.562509 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d34bed32-0175-4cd1-a742-7d0805d2349a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:19.589284 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:47:19.661133 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1064/2113] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:47:19 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:47:19.672081 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0ab6f4-f24f-4ee0-ae05-1384f4a0a813 service neutron] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:19.673561 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-5f0ab6f4-f24f-4ee0-ae05-1384f4a0a813 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:19.767597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0ab6f4-f24f-4ee0-ae05-1384f4a0a813 service neutron] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:19.771687 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe2c5a0-ff86-47ae-be6a-a110b6a9d56f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:19.772299 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe2c5a0-ff86-47ae-be6a-a110b6a9d56f service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:19.772762 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe2c5a0-ff86-47ae-be6a-a110b6a9d56f service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:19.773420 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe2c5a0-ff86-47ae-be6a-a110b6a9d56f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:19.786884 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe2c5a0-ff86-47ae-be6a-a110b6a9d56f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:19.788705 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe2c5a0-ff86-47ae-be6a-a110b6a9d56f service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:19.799210 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1050/2114] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:47:19 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:47:20.507419 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d34bed32-0175-4cd1-a742-7d0805d2349a None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:20.508948 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d34bed32-0175-4cd1-a742-7d0805d2349a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:20.523333 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d34bed32-0175-4cd1-a742-7d0805d2349a None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:20.528410 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c6a494-088f-440e-ad6a-19243e050448 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:20.529138 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c6a494-088f-440e-ad6a-19243e050448 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:20.529326 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c6a494-088f-440e-ad6a-19243e050448 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:20.529950 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c6a494-088f-440e-ad6a-19243e050448 admin admin] PATH_INFO: `/v3/users/9a9a75d645ce406ba22561e2f2b7b2cf` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:20.531127 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22c6a494-088f-440e-ad6a-19243e050448 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9a9a75d645ce406ba22561e2f2b7b2cf)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:20.533089 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22c6a494-088f-440e-ad6a-19243e050448 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:20.586864 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-22c6a494-088f-440e-ad6a-19243e050448 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9a9a75d645ce406ba22561e2f2b7b2cf'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:20.595047 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-22c6a494-088f-440e-ad6a-19243e050448 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9a9a75d645ce406ba22561e2f2b7b2cf'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:20.595774 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-22c6a494-088f-440e-ad6a-19243e050448 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9a9a75d645ce406ba22561e2f2b7b2cf'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:20.596379 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-22c6a494-088f-440e-ad6a-19243e050448 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9a9a75d645ce406ba22561e2f2b7b2cf'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:20.604405 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-22c6a494-088f-440e-ad6a-19243e050448 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9a9a75d645ce406ba22561e2f2b7b2cf'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:20.623033 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1065/2115] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:47:20 2022] DELETE /identity/v3/users/9a9a75d645ce406ba22561e2f2b7b2cf => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:47:20.627561 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d662c59c-1b06-48ea-ac21-9ca9bfc38ea4 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:20.627852 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d662c59c-1b06-48ea-ac21-9ca9bfc38ea4 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:20.628139 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d662c59c-1b06-48ea-ac21-9ca9bfc38ea4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:20.647581 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-d662c59c-1b06-48ea-ac21-9ca9bfc38ea4 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:47:20.648906 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-d662c59c-1b06-48ea-ac21-9ca9bfc38ea4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:20.708382 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1051/2116] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:47:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:47:20.718156 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c6a494-088f-440e-ad6a-19243e050448 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:20.719502 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-22c6a494-088f-440e-ad6a-19243e050448 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:20.776069 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c6a494-088f-440e-ad6a-19243e050448 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:20.780341 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71718fb0-5d1f-455a-9137-06e9b4b97d35 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:20.781052 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71718fb0-5d1f-455a-9137-06e9b4b97d35 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:20.781449 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71718fb0-5d1f-455a-9137-06e9b4b97d35 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:20.781731 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71718fb0-5d1f-455a-9137-06e9b4b97d35 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:20.791841 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71718fb0-5d1f-455a-9137-06e9b4b97d35 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:20.793519 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71718fb0-5d1f-455a-9137-06e9b4b97d35 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:20.803464 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1066/2117] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:47:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:47:21.087142 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d662c59c-1b06-48ea-ac21-9ca9bfc38ea4 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:21.089658 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-d662c59c-1b06-48ea-ac21-9ca9bfc38ea4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:21.104297 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d662c59c-1b06-48ea-ac21-9ca9bfc38ea4 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:21.110221 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4178fa31-0ff9-42ce-9ad5-3dd985810823 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:21.111182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4178fa31-0ff9-42ce-9ad5-3dd985810823 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:21.111684 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4178fa31-0ff9-42ce-9ad5-3dd985810823 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:21.112225 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4178fa31-0ff9-42ce-9ad5-3dd985810823 admin admin] PATH_INFO: `/v3/projects/aa635ba0a62a4806af2a9fac99e19a02` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:21.114687 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4178fa31-0ff9-42ce-9ad5-3dd985810823 admin admin] RBAC: Authorizing `identity:delete_project(project_id=aa635ba0a62a4806af2a9fac99e19a02)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:21.118479 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4178fa31-0ff9-42ce-9ad5-3dd985810823 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:21.175592 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-4178fa31-0ff9-42ce-9ad5-3dd985810823 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'aa635ba0a62a4806af2a9fac99e19a02', 'request_id': 'req-4178fa31-0ff9-42ce-9ad5-3dd985810823'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:21.183217 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-4178fa31-0ff9-42ce-9ad5-3dd985810823 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'aa635ba0a62a4806af2a9fac99e19a02', 'request_id': 'req-4178fa31-0ff9-42ce-9ad5-3dd985810823'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:21.193846 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-4178fa31-0ff9-42ce-9ad5-3dd985810823 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'aa635ba0a62a4806af2a9fac99e19a02', 'request_id': 'req-4178fa31-0ff9-42ce-9ad5-3dd985810823'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:21.195884 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:47:21.217568 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-4178fa31-0ff9-42ce-9ad5-3dd985810823 admin admin] The token cache is being invalidate because project aa635ba0a62a4806af2a9fac99e19a02 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:47:21.217568 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-4178fa31-0ff9-42ce-9ad5-3dd985810823 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:21.217568 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-4178fa31-0ff9-42ce-9ad5-3dd985810823 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:21.221550 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1052/2118] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:47:21 2022] DELETE /identity/v3/projects/aa635ba0a62a4806af2a9fac99e19a02 => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:47:21.223980 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98382bcd-4066-4896-879b-84188d81b55d None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:21.224655 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98382bcd-4066-4896-879b-84188d81b55d None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:21.225034 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98382bcd-4066-4896-879b-84188d81b55d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:21.248636 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-98382bcd-4066-4896-879b-84188d81b55d None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:47:21.250061 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-98382bcd-4066-4896-879b-84188d81b55d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:21.260958 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41675a45-ab5b-4ce7-b5de-55f9e4203dd8 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:21.262286 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41675a45-ab5b-4ce7-b5de-55f9e4203dd8 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:21.263275 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41675a45-ab5b-4ce7-b5de-55f9e4203dd8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:21.292682 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:47:21.294867 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-41675a45-ab5b-4ce7-b5de-55f9e4203dd8 None None] MFA Rules not processed for user `ec23c6593e7a446298d607294997664c`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:47:21.297798 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-41675a45-ab5b-4ce7-b5de-55f9e4203dd8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:21.340596 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1067/2119] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:47:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:47:21.348078 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1897a61-8576-4705-92f5-ceb89066fd3e None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:21.348364 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1897a61-8576-4705-92f5-ceb89066fd3e None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:21.348639 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1897a61-8576-4705-92f5-ceb89066fd3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:21.369881 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-d1897a61-8576-4705-92f5-ceb89066fd3e None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:47:21.371107 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d1897a61-8576-4705-92f5-ceb89066fd3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:21.380165 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1068/2120] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:47:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:47:21.382253 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1053/2121] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:47:21 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:47:21.384258 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1897a61-8576-4705-92f5-ceb89066fd3e None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:21.396068 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1897a61-8576-4705-92f5-ceb89066fd3e None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:21.401583 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121b26b5-476c-4d92-938d-2ba0eb02e15d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:21.402269 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121b26b5-476c-4d92-938d-2ba0eb02e15d admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:21.402649 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121b26b5-476c-4d92-938d-2ba0eb02e15d admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:21.402983 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121b26b5-476c-4d92-938d-2ba0eb02e15d admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:21.403862 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-121b26b5-476c-4d92-938d-2ba0eb02e15d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:47:21.404326 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-121b26b5-476c-4d92-938d-2ba0eb02e15d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:21.406734 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-121b26b5-476c-4d92-938d-2ba0eb02e15d admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:21.424133 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1069/2122] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:47:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:47:21.427633 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41675a45-ab5b-4ce7-b5de-55f9e4203dd8 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:21.446200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41675a45-ab5b-4ce7-b5de-55f9e4203dd8 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:21.459224 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4095d7fe-f4fd-4669-8e2a-7e6767180bf6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:21.460092 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4095d7fe-f4fd-4669-8e2a-7e6767180bf6 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:21.460554 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4095d7fe-f4fd-4669-8e2a-7e6767180bf6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:21.461024 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4095d7fe-f4fd-4669-8e2a-7e6767180bf6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:21.461853 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4095d7fe-f4fd-4669-8e2a-7e6767180bf6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:21.465050 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4095d7fe-f4fd-4669-8e2a-7e6767180bf6 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:21.496886 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1054/2123] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:47:21 2022] POST /identity/v3/projects => generated 375 bytes in 70 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:47:21.501475 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121b26b5-476c-4d92-938d-2ba0eb02e15d admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:21.513794 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121b26b5-476c-4d92-938d-2ba0eb02e15d admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:21.588824 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d0cc54-ffe3-4e58-9dc8-51a377934297 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:21.589447 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d0cc54-ffe3-4e58-9dc8-51a377934297 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:21.589678 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d0cc54-ffe3-4e58-9dc8-51a377934297 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:21.590134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d0cc54-ffe3-4e58-9dc8-51a377934297 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:21.590979 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d0cc54-ffe3-4e58-9dc8-51a377934297 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:21.593195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d0cc54-ffe3-4e58-9dc8-51a377934297 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:21.617173 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1070/2124] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:47:21 2022] POST /identity/v3/users => generated 288 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:47:21.621789 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4095d7fe-f4fd-4669-8e2a-7e6767180bf6 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:21.634225 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4095d7fe-f4fd-4669-8e2a-7e6767180bf6 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:21.639258 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6f32c9-bc8b-4b8c-8840-acf32293fb75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:21.639902 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6f32c9-bc8b-4b8c-8840-acf32293fb75 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:21.640187 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6f32c9-bc8b-4b8c-8840-acf32293fb75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:21.640470 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6f32c9-bc8b-4b8c-8840-acf32293fb75 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:21.641157 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa6f32c9-bc8b-4b8c-8840-acf32293fb75 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:21.642818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa6f32c9-bc8b-4b8c-8840-acf32293fb75 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:21.656333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1055/2125] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:47:21 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:47:21.657578 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d0cc54-ffe3-4e58-9dc8-51a377934297 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:21.669261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d0cc54-ffe3-4e58-9dc8-51a377934297 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:21.674305 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c5d7a8-fb33-46e6-bb6e-d3417b3055f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:21.675055 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c5d7a8-fb33-46e6-bb6e-d3417b3055f0 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:21.675420 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c5d7a8-fb33-46e6-bb6e-d3417b3055f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:21.676108 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c5d7a8-fb33-46e6-bb6e-d3417b3055f0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:21.676578 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44c5d7a8-fb33-46e6-bb6e-d3417b3055f0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:21.678919 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44c5d7a8-fb33-46e6-bb6e-d3417b3055f0 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:21.694030 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1071/2126] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:47:21 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:47:21.707369 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6f32c9-bc8b-4b8c-8840-acf32293fb75 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:21.717725 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6f32c9-bc8b-4b8c-8840-acf32293fb75 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:21.723786 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47cb4e14-e2b3-4a8c-b900-b608fb092f80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:21.724600 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47cb4e14-e2b3-4a8c-b900-b608fb092f80 admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:21.724886 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47cb4e14-e2b3-4a8c-b900-b608fb092f80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:21.725360 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47cb4e14-e2b3-4a8c-b900-b608fb092f80 admin admin] PATH_INFO: `/v3/projects/56e2dd956fdf483ba2965edef6e4f683/users/426c34be05a8465286b4292c132ea740/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:21.746860 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47cb4e14-e2b3-4a8c-b900-b608fb092f80 admin admin] RBAC: Authorizing `identity:create_grant(project_id=56e2dd956fdf483ba2965edef6e4f683, user_id=426c34be05a8465286b4292c132ea740, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:21.749635 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47cb4e14-e2b3-4a8c-b900-b608fb092f80 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:21.764747 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1056/2127] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:47:21 2022] PUT /identity/v3/projects/56e2dd956fdf483ba2965edef6e4f683/users/426c34be05a8465286b4292c132ea740/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:47:21.776392 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0309febf-d035-4cac-8b7f-924931ca81cf None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:21.776936 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0309febf-d035-4cac-8b7f-924931ca81cf None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:21.777217 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0309febf-d035-4cac-8b7f-924931ca81cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:21.808820 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-0309febf-d035-4cac-8b7f-924931ca81cf None None] MFA Rules not processed for user `426c34be05a8465286b4292c132ea740`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:47:21.810444 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-0309febf-d035-4cac-8b7f-924931ca81cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:21.954270 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1072/2128] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:47:21 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:47:21.958815 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47cb4e14-e2b3-4a8c-b900-b608fb092f80 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:22.025340 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47cb4e14-e2b3-4a8c-b900-b608fb092f80 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:22.030674 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c0380b-a487-4805-ac97-80ef82b4316a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:22.031284 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c0380b-a487-4805-ac97-80ef82b4316a admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:22.031565 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c0380b-a487-4805-ac97-80ef82b4316a admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:22.031839 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c0380b-a487-4805-ac97-80ef82b4316a admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:22.032486 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c0380b-a487-4805-ac97-80ef82b4316a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:22.034422 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c0380b-a487-4805-ac97-80ef82b4316a admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:22.065785 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1057/2129] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:47:21 2022] POST /identity/v3/projects => generated 375 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:47:22.070155 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0309febf-d035-4cac-8b7f-924931ca81cf None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:22.081045 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0309febf-d035-4cac-8b7f-924931ca81cf None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:22.102644 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:47:22.155833 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86aeb83f-6a13-4b32-af78-76c036e50e6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:22.156536 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86aeb83f-6a13-4b32-af78-76c036e50e6f admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:22.156912 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86aeb83f-6a13-4b32-af78-76c036e50e6f admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:22.157385 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86aeb83f-6a13-4b32-af78-76c036e50e6f admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:22.158001 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86aeb83f-6a13-4b32-af78-76c036e50e6f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:22.160874 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86aeb83f-6a13-4b32-af78-76c036e50e6f admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:22.211992 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1073/2130] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:47:22 2022] POST /identity/v3/users => generated 288 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:47:22.213245 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c0380b-a487-4805-ac97-80ef82b4316a admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:22.226730 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c0380b-a487-4805-ac97-80ef82b4316a admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:22.231632 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87997c9e-028c-4680-9ffe-eb7d87c716a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:22.232336 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87997c9e-028c-4680-9ffe-eb7d87c716a6 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:22.232696 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87997c9e-028c-4680-9ffe-eb7d87c716a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:22.233703 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87997c9e-028c-4680-9ffe-eb7d87c716a6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:22.234471 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87997c9e-028c-4680-9ffe-eb7d87c716a6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:22.236300 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87997c9e-028c-4680-9ffe-eb7d87c716a6 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:22.251276 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1058/2131] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:47:22 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:47:22.256287 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86aeb83f-6a13-4b32-af78-76c036e50e6f admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:22.280762 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86aeb83f-6a13-4b32-af78-76c036e50e6f admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:22.286129 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be28a352-4f38-4ebf-93ca-b6746666c672 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:22.287270 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be28a352-4f38-4ebf-93ca-b6746666c672 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:22.287537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be28a352-4f38-4ebf-93ca-b6746666c672 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:22.288791 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be28a352-4f38-4ebf-93ca-b6746666c672 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:22.289452 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be28a352-4f38-4ebf-93ca-b6746666c672 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:22.291491 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be28a352-4f38-4ebf-93ca-b6746666c672 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:22.304165 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1074/2132] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:47:22 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:47:22.308386 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87997c9e-028c-4680-9ffe-eb7d87c716a6 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:22.319286 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87997c9e-028c-4680-9ffe-eb7d87c716a6 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:22.324440 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2eba75-2cbf-49d9-ab58-39a06df2fad7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:22.325272 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2eba75-2cbf-49d9-ab58-39a06df2fad7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:22.325548 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2eba75-2cbf-49d9-ab58-39a06df2fad7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:22.325819 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2eba75-2cbf-49d9-ab58-39a06df2fad7 admin admin] PATH_INFO: `/v3/projects/062b168e301f49278b380fd52970e73f/users/6b24980d17424f4c8a0c6546d11ed0cf/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:22.335984 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:DELETE=2,keystone:SELECT=95,keystone:INSERT=3 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:47:22.345886 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c2eba75-2cbf-49d9-ab58-39a06df2fad7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=062b168e301f49278b380fd52970e73f, user_id=6b24980d17424f4c8a0c6546d11ed0cf, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:22.350980 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c2eba75-2cbf-49d9-ab58-39a06df2fad7 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:22.373474 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1059/2133] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:47:22 2022] PUT /identity/v3/projects/062b168e301f49278b380fd52970e73f/users/6b24980d17424f4c8a0c6546d11ed0cf/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:47:22.383931 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f1092e-9bde-40eb-a8e9-145a3bc45156 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:22.384236 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f1092e-9bde-40eb-a8e9-145a3bc45156 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:22.384495 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f1092e-9bde-40eb-a8e9-145a3bc45156 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:22.427339 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-24f1092e-9bde-40eb-a8e9-145a3bc45156 None None] MFA Rules not processed for user `6b24980d17424f4c8a0c6546d11ed0cf`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:47:22.428658 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-24f1092e-9bde-40eb-a8e9-145a3bc45156 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:22.527466 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1075/2134] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:47:22 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:47:22.531929 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2eba75-2cbf-49d9-ab58-39a06df2fad7 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:22.653690 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2eba75-2cbf-49d9-ab58-39a06df2fad7 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:22.659815 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c299ba8c-3c8d-4fa7-9f2e-8eaed2b3fd9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:22.661343 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c299ba8c-3c8d-4fa7-9f2e-8eaed2b3fd9d admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:22.661695 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c299ba8c-3c8d-4fa7-9f2e-8eaed2b3fd9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:22.662098 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c299ba8c-3c8d-4fa7-9f2e-8eaed2b3fd9d admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:22.662999 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c299ba8c-3c8d-4fa7-9f2e-8eaed2b3fd9d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:22.665633 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c299ba8c-3c8d-4fa7-9f2e-8eaed2b3fd9d admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:22.712293 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1060/2135] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:47:22 2022] POST /identity/v3/projects => generated 377 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:47:22.713403 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f1092e-9bde-40eb-a8e9-145a3bc45156 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:22.738305 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f1092e-9bde-40eb-a8e9-145a3bc45156 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:22.822013 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13355b09-9787-4953-9c6f-7d4f3fbcca13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:22.822680 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13355b09-9787-4953-9c6f-7d4f3fbcca13 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:22.823017 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13355b09-9787-4953-9c6f-7d4f3fbcca13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:22.823296 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13355b09-9787-4953-9c6f-7d4f3fbcca13 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:22.824056 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13355b09-9787-4953-9c6f-7d4f3fbcca13 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:22.826255 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13355b09-9787-4953-9c6f-7d4f3fbcca13 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:22.863514 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1076/2136] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:47:22 2022] POST /identity/v3/users => generated 295 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:47:22.866624 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c299ba8c-3c8d-4fa7-9f2e-8eaed2b3fd9d admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:22.891899 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c299ba8c-3c8d-4fa7-9f2e-8eaed2b3fd9d admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:22.901784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b52d4d-4e62-4b41-be1f-3cf5787068fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:22.902583 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b52d4d-4e62-4b41-be1f-3cf5787068fc admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:22.903443 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b52d4d-4e62-4b41-be1f-3cf5787068fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:22.904029 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b52d4d-4e62-4b41-be1f-3cf5787068fc admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:22.904856 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26b52d4d-4e62-4b41-be1f-3cf5787068fc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:22.907100 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26b52d4d-4e62-4b41-be1f-3cf5787068fc admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:22.920967 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1061/2137] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:47:22 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:47:22.926837 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13355b09-9787-4953-9c6f-7d4f3fbcca13 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:22.938135 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13355b09-9787-4953-9c6f-7d4f3fbcca13 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:22.943360 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f0bb2b-71ef-401f-b727-9a143d6ae607 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:22.944098 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f0bb2b-71ef-401f-b727-9a143d6ae607 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:22.944413 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f0bb2b-71ef-401f-b727-9a143d6ae607 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:22.944812 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f0bb2b-71ef-401f-b727-9a143d6ae607 admin admin] PATH_INFO: `/v3/projects/4bc09ce339114d448dc1b45fc73877df/users/0db352e6a126410198d585abaceabcac/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:22.962671 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8f0bb2b-71ef-401f-b727-9a143d6ae607 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4bc09ce339114d448dc1b45fc73877df, user_id=0db352e6a126410198d585abaceabcac, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:22.966016 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8f0bb2b-71ef-401f-b727-9a143d6ae607 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:22.970044 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:47:22.992949 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1077/2138] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:47:22 2022] PUT /identity/v3/projects/4bc09ce339114d448dc1b45fc73877df/users/0db352e6a126410198d585abaceabcac/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 68 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:47:22.998138 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b52d4d-4e62-4b41-be1f-3cf5787068fc admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:23.037827 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b52d4d-4e62-4b41-be1f-3cf5787068fc admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:23.151426 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce05b838-41d6-4e4d-80c2-4b8ec1180b78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:23.152583 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce05b838-41d6-4e4d-80c2-4b8ec1180b78 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:23.153160 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce05b838-41d6-4e4d-80c2-4b8ec1180b78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:23.153790 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce05b838-41d6-4e4d-80c2-4b8ec1180b78 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:23.154828 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce05b838-41d6-4e4d-80c2-4b8ec1180b78 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:23.156997 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce05b838-41d6-4e4d-80c2-4b8ec1180b78 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:23.171561 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1062/2139] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:47:22 2022] GET /identity/v3/roles => generated 1420 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:47:23.177427 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f0bb2b-71ef-401f-b727-9a143d6ae607 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:23.196319 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f0bb2b-71ef-401f-b727-9a143d6ae607 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:23.203866 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6f8ddd-6edc-41f2-8d32-4156f56c0a6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:23.205081 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6f8ddd-6edc-41f2-8d32-4156f56c0a6b admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:23.205619 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6f8ddd-6edc-41f2-8d32-4156f56c0a6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:23.206460 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6f8ddd-6edc-41f2-8d32-4156f56c0a6b admin admin] PATH_INFO: `/v3/projects/4bc09ce339114d448dc1b45fc73877df/users/0db352e6a126410198d585abaceabcac/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:23.209887 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b6f8ddd-6edc-41f2-8d32-4156f56c0a6b admin admin] RBAC: Authorizing `identity:create_grant(project_id=4bc09ce339114d448dc1b45fc73877df, user_id=0db352e6a126410198d585abaceabcac, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:23.212801 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b6f8ddd-6edc-41f2-8d32-4156f56c0a6b admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:23.225115 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1078/2140] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:47:23 2022] PUT /identity/v3/projects/4bc09ce339114d448dc1b45fc73877df/users/0db352e6a126410198d585abaceabcac/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:47:23.237451 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6fea7ee-1b31-4331-a44d-0cd30dd06dfa None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:23.237738 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6fea7ee-1b31-4331-a44d-0cd30dd06dfa None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:23.240104 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6fea7ee-1b31-4331-a44d-0cd30dd06dfa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:23.289967 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-a6fea7ee-1b31-4331-a44d-0cd30dd06dfa None None] MFA Rules not processed for user `0db352e6a126410198d585abaceabcac`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:47:23.292151 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-a6fea7ee-1b31-4331-a44d-0cd30dd06dfa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:23.356718 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:47:23.392688 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1063/2141] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:47:23 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:47:23.406122 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6f8ddd-6edc-41f2-8d32-4156f56c0a6b admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:23.407908 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-3b6f8ddd-6edc-41f2-8d32-4156f56c0a6b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:23.486958 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6f8ddd-6edc-41f2-8d32-4156f56c0a6b admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:23.490779 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c0e7fbb-4be3-4fd7-bd97-499f602b0e30 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:23.491400 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0e7fbb-4be3-4fd7-bd97-499f602b0e30 service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:23.491685 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0e7fbb-4be3-4fd7-bd97-499f602b0e30 service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:23.491962 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0e7fbb-4be3-4fd7-bd97-499f602b0e30 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:23.563187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c0e7fbb-4be3-4fd7-bd97-499f602b0e30 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:23.564602 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c0e7fbb-4be3-4fd7-bd97-499f602b0e30 service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:23.575327 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1079/2142] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:47:23 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 170 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:47:25.132874 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6fea7ee-1b31-4331-a44d-0cd30dd06dfa None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:25.134354 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-a6fea7ee-1b31-4331-a44d-0cd30dd06dfa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:25.148082 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6fea7ee-1b31-4331-a44d-0cd30dd06dfa None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:25.151940 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7154c0a-1044-45be-8e15-faf04dd349f6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:25.152650 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7154c0a-1044-45be-8e15-faf04dd349f6 service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:25.153031 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7154c0a-1044-45be-8e15-faf04dd349f6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:25.153429 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7154c0a-1044-45be-8e15-faf04dd349f6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:25.217989 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7154c0a-1044-45be-8e15-faf04dd349f6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:25.219503 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7154c0a-1044-45be-8e15-faf04dd349f6 service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:25.242567 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1064/2143] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:47:25 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:47:25.865189 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c0e7fbb-4be3-4fd7-bd97-499f602b0e30 service cinder] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:25.873697 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c0e7fbb-4be3-4fd7-bd97-499f602b0e30 service cinder] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:25.877636 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c98ff2-70fc-4564-8ddf-e51a34183122 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:25.878240 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c98ff2-70fc-4564-8ddf-e51a34183122 service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:25.878573 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c98ff2-70fc-4564-8ddf-e51a34183122 service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:25.878851 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c98ff2-70fc-4564-8ddf-e51a34183122 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:25.892306 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c98ff2-70fc-4564-8ddf-e51a34183122 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:25.894012 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c98ff2-70fc-4564-8ddf-e51a34183122 service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:25.905138 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1080/2144] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:47:25 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:47:28.682120 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7154c0a-1044-45be-8e15-faf04dd349f6 service cinder] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:28.683888 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-a7154c0a-1044-45be-8e15-faf04dd349f6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:28.815087 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7154c0a-1044-45be-8e15-faf04dd349f6 service cinder] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:28.819819 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa5496d8-d9b6-4015-9a57-774f30760d46 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:28.820631 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa5496d8-d9b6-4015-9a57-774f30760d46 service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:28.820964 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa5496d8-d9b6-4015-9a57-774f30760d46 service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:28.821297 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa5496d8-d9b6-4015-9a57-774f30760d46 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:28.825340 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-fa5496d8-d9b6-4015-9a57-774f30760d46 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:28.915957 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa5496d8-d9b6-4015-9a57-774f30760d46 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:28.917787 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa5496d8-d9b6-4015-9a57-774f30760d46 service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:28.929893 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1065/2145] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:47:28 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 248 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:47:29.820401 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d422c1-2067-4ba6-9a7b-0760d811a9c5 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:29.821008 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d422c1-2067-4ba6-9a7b-0760d811a9c5 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:29.821008 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d422c1-2067-4ba6-9a7b-0760d811a9c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:29.842942 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-80d422c1-2067-4ba6-9a7b-0760d811a9c5 None None] MFA Rules not processed for user `ed9a4c24cbfb4689b7f984219c7503a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:47:29.844236 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-80d422c1-2067-4ba6-9a7b-0760d811a9c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:29.909551 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1081/2146] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:47:29 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:47:29.990420 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa5496d8-d9b6-4015-9a57-774f30760d46 service neutron] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:30.026694 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:47:30.067571 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa5496d8-d9b6-4015-9a57-774f30760d46 service neutron] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:30.072572 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf815b35-1c65-489a-856c-7aee57276f57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:30.073397 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf815b35-1c65-489a-856c-7aee57276f57 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:30.073860 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf815b35-1c65-489a-856c-7aee57276f57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:30.074365 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf815b35-1c65-489a-856c-7aee57276f57 admin admin] PATH_INFO: `/v3/users/426c34be05a8465286b4292c132ea740` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:30.075744 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf815b35-1c65-489a-856c-7aee57276f57 admin admin] RBAC: Authorizing `identity:delete_user(user_id=426c34be05a8465286b4292c132ea740)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:30.077717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf815b35-1c65-489a-856c-7aee57276f57 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:30.141367 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-bf815b35-1c65-489a-856c-7aee57276f57 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '426c34be05a8465286b4292c132ea740'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:30.151319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-bf815b35-1c65-489a-856c-7aee57276f57 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '426c34be05a8465286b4292c132ea740'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:30.152003 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-bf815b35-1c65-489a-856c-7aee57276f57 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '426c34be05a8465286b4292c132ea740'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:30.152684 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-bf815b35-1c65-489a-856c-7aee57276f57 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '426c34be05a8465286b4292c132ea740'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:30.162310 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-bf815b35-1c65-489a-856c-7aee57276f57 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '426c34be05a8465286b4292c132ea740'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:30.185203 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1066/2147] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:47:29 2022] DELETE /identity/v3/users/426c34be05a8465286b4292c132ea740 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:47:30.190005 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4384a89a-0aa7-4c2c-8f07-4d65b17051e4 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:30.190838 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4384a89a-0aa7-4c2c-8f07-4d65b17051e4 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:30.192089 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4384a89a-0aa7-4c2c-8f07-4d65b17051e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:30.217420 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-4384a89a-0aa7-4c2c-8f07-4d65b17051e4 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:47:30.219053 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4384a89a-0aa7-4c2c-8f07-4d65b17051e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:30.247408 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:47:30.297744 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1082/2148] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:47:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:47:30.332135 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf815b35-1c65-489a-856c-7aee57276f57 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:30.337554 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-bf815b35-1c65-489a-856c-7aee57276f57 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:30.420682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf815b35-1c65-489a-856c-7aee57276f57 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:30.426010 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85468412-0bcd-40bc-88da-4b6693f922d4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:30.426667 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85468412-0bcd-40bc-88da-4b6693f922d4 service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:30.427083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85468412-0bcd-40bc-88da-4b6693f922d4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:30.427374 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85468412-0bcd-40bc-88da-4b6693f922d4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:30.438449 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85468412-0bcd-40bc-88da-4b6693f922d4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:30.440782 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85468412-0bcd-40bc-88da-4b6693f922d4 service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:30.451897 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1067/2149] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:47:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:47:31.501976 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4384a89a-0aa7-4c2c-8f07-4d65b17051e4 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:31.504010 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4384a89a-0aa7-4c2c-8f07-4d65b17051e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:31.516962 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4384a89a-0aa7-4c2c-8f07-4d65b17051e4 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:31.526430 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7380a34a-fc12-4b75-b8c9-59cbaae2a6d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:31.527423 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7380a34a-fc12-4b75-b8c9-59cbaae2a6d4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:31.527917 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7380a34a-fc12-4b75-b8c9-59cbaae2a6d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:31.528431 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7380a34a-fc12-4b75-b8c9-59cbaae2a6d4 admin admin] PATH_INFO: `/v3/projects/56e2dd956fdf483ba2965edef6e4f683` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:31.531721 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7380a34a-fc12-4b75-b8c9-59cbaae2a6d4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=56e2dd956fdf483ba2965edef6e4f683)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:31.536062 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7380a34a-fc12-4b75-b8c9-59cbaae2a6d4 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:31.671306 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7380a34a-fc12-4b75-b8c9-59cbaae2a6d4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '56e2dd956fdf483ba2965edef6e4f683', 'request_id': 'req-7380a34a-fc12-4b75-b8c9-59cbaae2a6d4'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:31.688175 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7380a34a-fc12-4b75-b8c9-59cbaae2a6d4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '56e2dd956fdf483ba2965edef6e4f683', 'request_id': 'req-7380a34a-fc12-4b75-b8c9-59cbaae2a6d4'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:31.703103 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7380a34a-fc12-4b75-b8c9-59cbaae2a6d4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '56e2dd956fdf483ba2965edef6e4f683', 'request_id': 'req-7380a34a-fc12-4b75-b8c9-59cbaae2a6d4'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:31.732107 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7380a34a-fc12-4b75-b8c9-59cbaae2a6d4 admin admin] The token cache is being invalidate because project 56e2dd956fdf483ba2965edef6e4f683 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:47:31.732429 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7380a34a-fc12-4b75-b8c9-59cbaae2a6d4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:31.735639 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7380a34a-fc12-4b75-b8c9-59cbaae2a6d4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:31.739554 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1083/2150] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:47:31 2022] DELETE /identity/v3/projects/56e2dd956fdf483ba2965edef6e4f683 => generated 0 bytes in 238 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:47:31.744829 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85468412-0bcd-40bc-88da-4b6693f922d4 service neutron] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:31.757857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85468412-0bcd-40bc-88da-4b6693f922d4 service neutron] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:31.876130 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:47:31.899245 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e251922d-237d-4432-9a15-92a79ee7e862 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:31.899908 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e251922d-237d-4432-9a15-92a79ee7e862 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:31.900183 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e251922d-237d-4432-9a15-92a79ee7e862 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:31.900581 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e251922d-237d-4432-9a15-92a79ee7e862 admin admin] PATH_INFO: `/v3/users/6b24980d17424f4c8a0c6546d11ed0cf` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:31.903750 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e251922d-237d-4432-9a15-92a79ee7e862 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6b24980d17424f4c8a0c6546d11ed0cf)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:31.905992 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e251922d-237d-4432-9a15-92a79ee7e862 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:32.025779 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-e251922d-237d-4432-9a15-92a79ee7e862 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6b24980d17424f4c8a0c6546d11ed0cf'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:32.045248 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-e251922d-237d-4432-9a15-92a79ee7e862 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6b24980d17424f4c8a0c6546d11ed0cf'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:32.046157 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-e251922d-237d-4432-9a15-92a79ee7e862 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6b24980d17424f4c8a0c6546d11ed0cf'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:32.047261 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-e251922d-237d-4432-9a15-92a79ee7e862 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6b24980d17424f4c8a0c6546d11ed0cf'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:32.058435 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-e251922d-237d-4432-9a15-92a79ee7e862 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6b24980d17424f4c8a0c6546d11ed0cf'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:32.111528 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1068/2151] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:47:31 2022] DELETE /identity/v3/users/6b24980d17424f4c8a0c6546d11ed0cf => generated 0 bytes in 365 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:47:32.878296 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7380a34a-fc12-4b75-b8c9-59cbaae2a6d4 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:32.880204 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7380a34a-fc12-4b75-b8c9-59cbaae2a6d4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:32.904136 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7380a34a-fc12-4b75-b8c9-59cbaae2a6d4 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:33.006987 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aed85a9-9648-4a8a-ad23-e22c08ab5e21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:33.007637 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aed85a9-9648-4a8a-ad23-e22c08ab5e21 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:33.007915 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aed85a9-9648-4a8a-ad23-e22c08ab5e21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:33.008191 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aed85a9-9648-4a8a-ad23-e22c08ab5e21 admin admin] PATH_INFO: `/v3/projects/062b168e301f49278b380fd52970e73f` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:33.009506 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aed85a9-9648-4a8a-ad23-e22c08ab5e21 admin admin] RBAC: Authorizing `identity:delete_project(project_id=062b168e301f49278b380fd52970e73f)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:33.011532 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aed85a9-9648-4a8a-ad23-e22c08ab5e21 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:33.067355 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-9aed85a9-9648-4a8a-ad23-e22c08ab5e21 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '062b168e301f49278b380fd52970e73f', 'request_id': 'req-9aed85a9-9648-4a8a-ad23-e22c08ab5e21'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:33.088983 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-9aed85a9-9648-4a8a-ad23-e22c08ab5e21 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '062b168e301f49278b380fd52970e73f', 'request_id': 'req-9aed85a9-9648-4a8a-ad23-e22c08ab5e21'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:33.091641 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:47:33.105869 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-9aed85a9-9648-4a8a-ad23-e22c08ab5e21 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '062b168e301f49278b380fd52970e73f', 'request_id': 'req-9aed85a9-9648-4a8a-ad23-e22c08ab5e21'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:33.125222 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-9aed85a9-9648-4a8a-ad23-e22c08ab5e21 admin admin] The token cache is being invalidate because project 062b168e301f49278b380fd52970e73f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:47:33.126191 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-9aed85a9-9648-4a8a-ad23-e22c08ab5e21 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:33.127086 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-9aed85a9-9648-4a8a-ad23-e22c08ab5e21 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:33.128825 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1084/2152] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:47:32 2022] DELETE /identity/v3/projects/062b168e301f49278b380fd52970e73f => generated 0 bytes in 251 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:47:33.133509 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e251922d-237d-4432-9a15-92a79ee7e862 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:33.135522 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-e251922d-237d-4432-9a15-92a79ee7e862 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:33.157059 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e251922d-237d-4432-9a15-92a79ee7e862 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:33.236655 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba17a351-ad1a-4707-943d-85d632eeb457 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:33.237302 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba17a351-ad1a-4707-943d-85d632eeb457 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:33.237585 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba17a351-ad1a-4707-943d-85d632eeb457 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:33.239130 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba17a351-ad1a-4707-943d-85d632eeb457 admin admin] PATH_INFO: `/v3/users/0db352e6a126410198d585abaceabcac` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:33.240171 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba17a351-ad1a-4707-943d-85d632eeb457 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0db352e6a126410198d585abaceabcac)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:33.242107 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba17a351-ad1a-4707-943d-85d632eeb457 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:33.291664 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-ba17a351-ad1a-4707-943d-85d632eeb457 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0db352e6a126410198d585abaceabcac'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:33.301429 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-ba17a351-ad1a-4707-943d-85d632eeb457 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0db352e6a126410198d585abaceabcac'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:33.302200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-ba17a351-ad1a-4707-943d-85d632eeb457 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0db352e6a126410198d585abaceabcac'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:33.302843 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-ba17a351-ad1a-4707-943d-85d632eeb457 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0db352e6a126410198d585abaceabcac'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:33.310528 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-ba17a351-ad1a-4707-943d-85d632eeb457 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0db352e6a126410198d585abaceabcac'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:33.318541 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=79,keystone:DELETE=19,keystone:INSERT=2 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:47:33.332673 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1069/2153] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:47:33 2022] DELETE /identity/v3/users/0db352e6a126410198d585abaceabcac => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:47:33.676094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aed85a9-9648-4a8a-ad23-e22c08ab5e21 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:33.678216 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-9aed85a9-9648-4a8a-ad23-e22c08ab5e21 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:33.698252 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aed85a9-9648-4a8a-ad23-e22c08ab5e21 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:33.765845 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c176d8-d1de-45ae-b5cc-dd4d107ce78c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:33.766539 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c176d8-d1de-45ae-b5cc-dd4d107ce78c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:33.766827 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c176d8-d1de-45ae-b5cc-dd4d107ce78c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:33.767491 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c176d8-d1de-45ae-b5cc-dd4d107ce78c admin admin] PATH_INFO: `/v3/projects/4bc09ce339114d448dc1b45fc73877df` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:33.768374 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51c176d8-d1de-45ae-b5cc-dd4d107ce78c admin admin] RBAC: Authorizing `identity:delete_project(project_id=4bc09ce339114d448dc1b45fc73877df)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:33.771125 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51c176d8-d1de-45ae-b5cc-dd4d107ce78c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:33.826333 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-51c176d8-d1de-45ae-b5cc-dd4d107ce78c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4bc09ce339114d448dc1b45fc73877df', 'request_id': 'req-51c176d8-d1de-45ae-b5cc-dd4d107ce78c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:33.833067 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-51c176d8-d1de-45ae-b5cc-dd4d107ce78c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4bc09ce339114d448dc1b45fc73877df', 'request_id': 'req-51c176d8-d1de-45ae-b5cc-dd4d107ce78c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:33.841150 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-51c176d8-d1de-45ae-b5cc-dd4d107ce78c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4bc09ce339114d448dc1b45fc73877df', 'request_id': 'req-51c176d8-d1de-45ae-b5cc-dd4d107ce78c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:33.856180 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-51c176d8-d1de-45ae-b5cc-dd4d107ce78c admin admin] The token cache is being invalidate because project 4bc09ce339114d448dc1b45fc73877df was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:47:33.856448 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-51c176d8-d1de-45ae-b5cc-dd4d107ce78c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:33.857128 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-51c176d8-d1de-45ae-b5cc-dd4d107ce78c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:33.858916 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1085/2154] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:47:33 2022] DELETE /identity/v3/projects/4bc09ce339114d448dc1b45fc73877df => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:47:33.863687 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb85916-5d64-440e-9fdf-b3000b5d6f60 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:33.863984 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb85916-5d64-440e-9fdf-b3000b5d6f60 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:33.864270 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb85916-5d64-440e-9fdf-b3000b5d6f60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:33.911420 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-3eb85916-5d64-440e-9fdf-b3000b5d6f60 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:47:33.912911 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-3eb85916-5d64-440e-9fdf-b3000b5d6f60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:33.990120 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1070/2155] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:47:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:47:33.997565 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9007ba9-5fc2-4341-b179-8b0524f20b62 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:33.997967 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9007ba9-5fc2-4341-b179-8b0524f20b62 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:33.998118 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9007ba9-5fc2-4341-b179-8b0524f20b62 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:34.023081 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-c9007ba9-5fc2-4341-b179-8b0524f20b62 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:47:34.024582 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-c9007ba9-5fc2-4341-b179-8b0524f20b62 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:34.032635 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1086/2156] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:47:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:47:34.037054 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb85916-5d64-440e-9fdf-b3000b5d6f60 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:34.047233 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb85916-5d64-440e-9fdf-b3000b5d6f60 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:34.054207 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-496ae432-eac7-49c7-9ba3-4cef788eea98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:34.054864 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496ae432-eac7-49c7-9ba3-4cef788eea98 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:34.055172 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496ae432-eac7-49c7-9ba3-4cef788eea98 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:34.055425 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-496ae432-eac7-49c7-9ba3-4cef788eea98 admin admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:34.056086 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-496ae432-eac7-49c7-9ba3-4cef788eea98 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:47:34.056396 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-496ae432-eac7-49c7-9ba3-4cef788eea98 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:34.058012 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-496ae432-eac7-49c7-9ba3-4cef788eea98 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:34.076110 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1071/2157] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:47:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:47:34.077426 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9007ba9-5fc2-4341-b179-8b0524f20b62 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:34.087672 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9007ba9-5fc2-4341-b179-8b0524f20b62 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:34.092477 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c9b0c3-7ec8-497c-85e7-8ddd280089f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:34.093090 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c9b0c3-7ec8-497c-85e7-8ddd280089f2 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:34.093375 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c9b0c3-7ec8-497c-85e7-8ddd280089f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:34.093671 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c9b0c3-7ec8-497c-85e7-8ddd280089f2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:34.094287 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4c9b0c3-7ec8-497c-85e7-8ddd280089f2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:34.096328 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4c9b0c3-7ec8-497c-85e7-8ddd280089f2 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:34.129862 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1087/2158] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:47:34 2022] POST /identity/v3/projects => generated 383 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:47:34.134148 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-496ae432-eac7-49c7-9ba3-4cef788eea98 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:34.144343 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-496ae432-eac7-49c7-9ba3-4cef788eea98 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:34.233916 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301e78f0-ffb8-401c-bccd-fe05c3f8e7f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:34.234725 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301e78f0-ffb8-401c-bccd-fe05c3f8e7f4 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:34.235250 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301e78f0-ffb8-401c-bccd-fe05c3f8e7f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:34.236148 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301e78f0-ffb8-401c-bccd-fe05c3f8e7f4 admin admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:34.237020 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-301e78f0-ffb8-401c-bccd-fe05c3f8e7f4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:34.240244 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-301e78f0-ffb8-401c-bccd-fe05c3f8e7f4 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:34.268701 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1072/2159] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:47:34 2022] POST /identity/v3/users => generated 292 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:47:34.272837 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c9b0c3-7ec8-497c-85e7-8ddd280089f2 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:34.286607 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4c9b0c3-7ec8-497c-85e7-8ddd280089f2 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:34.298236 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b035927d-25bb-49c5-abf3-02937ccbb7f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:34.298926 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b035927d-25bb-49c5-abf3-02937ccbb7f9 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:34.300021 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b035927d-25bb-49c5-abf3-02937ccbb7f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:34.301264 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b035927d-25bb-49c5-abf3-02937ccbb7f9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:34.303600 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b035927d-25bb-49c5-abf3-02937ccbb7f9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:34.306110 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b035927d-25bb-49c5-abf3-02937ccbb7f9 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:34.327008 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1088/2160] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:47:34 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:47:34.331052 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301e78f0-ffb8-401c-bccd-fe05c3f8e7f4 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:34.342729 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301e78f0-ffb8-401c-bccd-fe05c3f8e7f4 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:34.347850 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0825575f-7950-4e76-85bf-2b2acca10716 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:34.348903 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0825575f-7950-4e76-85bf-2b2acca10716 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:34.349268 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0825575f-7950-4e76-85bf-2b2acca10716 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:34.349740 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0825575f-7950-4e76-85bf-2b2acca10716 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:34.350535 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0825575f-7950-4e76-85bf-2b2acca10716 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:34.352317 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0825575f-7950-4e76-85bf-2b2acca10716 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:34.364457 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1073/2161] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:47:34 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:47:34.368960 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b035927d-25bb-49c5-abf3-02937ccbb7f9 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:34.379934 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b035927d-25bb-49c5-abf3-02937ccbb7f9 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:34.384765 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf71b49c-85a5-4153-bc47-aa112c9d617b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:34.385420 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf71b49c-85a5-4153-bc47-aa112c9d617b admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:34.385697 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf71b49c-85a5-4153-bc47-aa112c9d617b admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:34.385977 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf71b49c-85a5-4153-bc47-aa112c9d617b admin admin] PATH_INFO: `/v3/projects/585c44f8af1d43ed8d12a98e766fa7e9/users/2dc9e8a3fb4c4c79ad66f4a035e5931b/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:34.409185 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf71b49c-85a5-4153-bc47-aa112c9d617b admin admin] RBAC: Authorizing `identity:create_grant(project_id=585c44f8af1d43ed8d12a98e766fa7e9, user_id=2dc9e8a3fb4c4c79ad66f4a035e5931b, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:34.414698 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf71b49c-85a5-4153-bc47-aa112c9d617b admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:34.441430 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1089/2162] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:47:34 2022] PUT /identity/v3/projects/585c44f8af1d43ed8d12a98e766fa7e9/users/2dc9e8a3fb4c4c79ad66f4a035e5931b/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:47:34.444645 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f342ca-af3e-42a3-92a2-b9f43eb8774b None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:34.445000 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f342ca-af3e-42a3-92a2-b9f43eb8774b None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:34.445345 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f342ca-af3e-42a3-92a2-b9f43eb8774b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:34.476448 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-66f342ca-af3e-42a3-92a2-b9f43eb8774b None None] MFA Rules not processed for user `2dc9e8a3fb4c4c79ad66f4a035e5931b`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:47:34.477743 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-66f342ca-af3e-42a3-92a2-b9f43eb8774b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:34.540098 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=96,keystone:DELETE=1,keystone:INSERT=3 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:47:34.566949 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1074/2163] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:47:34 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:47:34.574427 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf71b49c-85a5-4153-bc47-aa112c9d617b admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:34.588127 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:INSERT=4,keystone:DELETE=10,keystone:SELECT=86 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:47:34.661697 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf71b49c-85a5-4153-bc47-aa112c9d617b admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:34.681201 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b17aaed-e596-4097-b565-5d78a55813b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:34.682436 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b17aaed-e596-4097-b565-5d78a55813b8 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:34.682768 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b17aaed-e596-4097-b565-5d78a55813b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:34.683050 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b17aaed-e596-4097-b565-5d78a55813b8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:34.683715 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b17aaed-e596-4097-b565-5d78a55813b8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:34.685618 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b17aaed-e596-4097-b565-5d78a55813b8 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:34.746648 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1090/2164] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:47:34 2022] POST /identity/v3/projects => generated 385 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:47:34.748082 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f342ca-af3e-42a3-92a2-b9f43eb8774b None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:34.770873 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f342ca-af3e-42a3-92a2-b9f43eb8774b None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:34.856719 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df7025d2-8fd0-42ce-b044-4ec813b2edfd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:34.857529 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7025d2-8fd0-42ce-b044-4ec813b2edfd admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:34.857875 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7025d2-8fd0-42ce-b044-4ec813b2edfd admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:34.858355 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7025d2-8fd0-42ce-b044-4ec813b2edfd admin admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:34.859729 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df7025d2-8fd0-42ce-b044-4ec813b2edfd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:34.864862 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df7025d2-8fd0-42ce-b044-4ec813b2edfd admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:34.894729 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1075/2165] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:47:34 2022] POST /identity/v3/users => generated 293 bytes in 147 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:47:34.895354 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b17aaed-e596-4097-b565-5d78a55813b8 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:34.909013 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b17aaed-e596-4097-b565-5d78a55813b8 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:34.926149 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8adc2b94-73f7-4cbb-9844-f04ffe8c5c21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:34.926825 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8adc2b94-73f7-4cbb-9844-f04ffe8c5c21 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:34.927107 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8adc2b94-73f7-4cbb-9844-f04ffe8c5c21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:34.927380 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8adc2b94-73f7-4cbb-9844-f04ffe8c5c21 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:34.928895 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8adc2b94-73f7-4cbb-9844-f04ffe8c5c21 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:34.930553 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8adc2b94-73f7-4cbb-9844-f04ffe8c5c21 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:34.949021 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1091/2166] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:47:34 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:47:34.953399 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df7025d2-8fd0-42ce-b044-4ec813b2edfd admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:34.970019 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df7025d2-8fd0-42ce-b044-4ec813b2edfd admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:34.977651 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d91c4fa7-a7fb-4cdd-9293-b8459ed178c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:34.979043 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d91c4fa7-a7fb-4cdd-9293-b8459ed178c9 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:34.979419 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d91c4fa7-a7fb-4cdd-9293-b8459ed178c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:34.979782 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d91c4fa7-a7fb-4cdd-9293-b8459ed178c9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:34.980539 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d91c4fa7-a7fb-4cdd-9293-b8459ed178c9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:34.982467 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d91c4fa7-a7fb-4cdd-9293-b8459ed178c9 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:35.014768 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1076/2167] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:47:34 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:47:35.024714 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8adc2b94-73f7-4cbb-9844-f04ffe8c5c21 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:35.060540 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8adc2b94-73f7-4cbb-9844-f04ffe8c5c21 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:35.066028 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4261189c-4bcb-48e3-ab88-abd7b54d9d27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:35.066778 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4261189c-4bcb-48e3-ab88-abd7b54d9d27 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:35.067219 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4261189c-4bcb-48e3-ab88-abd7b54d9d27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:35.067711 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4261189c-4bcb-48e3-ab88-abd7b54d9d27 admin admin] PATH_INFO: `/v3/projects/c4e99a697e494b02ab34f35d4a505e6d/users/6bac7605bd0043d69c496418c9a0d17e/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:35.088545 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4261189c-4bcb-48e3-ab88-abd7b54d9d27 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c4e99a697e494b02ab34f35d4a505e6d, user_id=6bac7605bd0043d69c496418c9a0d17e, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:35.092124 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4261189c-4bcb-48e3-ab88-abd7b54d9d27 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:35.113618 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1092/2168] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:47:35 2022] PUT /identity/v3/projects/c4e99a697e494b02ab34f35d4a505e6d/users/6bac7605bd0043d69c496418c9a0d17e/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 89 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:47:35.124765 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e916c9ed-1d8f-402b-9e73-894cb00bfd6a None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:35.125063 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e916c9ed-1d8f-402b-9e73-894cb00bfd6a None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:35.125393 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e916c9ed-1d8f-402b-9e73-894cb00bfd6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:35.163411 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-e916c9ed-1d8f-402b-9e73-894cb00bfd6a None None] MFA Rules not processed for user `6bac7605bd0043d69c496418c9a0d17e`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:47:35.168262 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-e916c9ed-1d8f-402b-9e73-894cb00bfd6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:35.323381 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1077/2169] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:47:35 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:47:35.325056 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4261189c-4bcb-48e3-ab88-abd7b54d9d27 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:35.408473 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4261189c-4bcb-48e3-ab88-abd7b54d9d27 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:35.416440 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9165213c-fdb7-401a-b385-9ed7af80cfd1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:35.417285 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9165213c-fdb7-401a-b385-9ed7af80cfd1 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:35.417285 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9165213c-fdb7-401a-b385-9ed7af80cfd1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:35.417838 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9165213c-fdb7-401a-b385-9ed7af80cfd1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:35.418569 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9165213c-fdb7-401a-b385-9ed7af80cfd1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:35.421227 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9165213c-fdb7-401a-b385-9ed7af80cfd1 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:35.481048 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1093/2170] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:47:35 2022] POST /identity/v3/projects => generated 383 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:47:35.485195 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e916c9ed-1d8f-402b-9e73-894cb00bfd6a None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:35.505726 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e916c9ed-1d8f-402b-9e73-894cb00bfd6a None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:35.687943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:47:35.716197 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cfa40aa-f4c3-4f65-9ab1-75fc3422c3a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:35.717536 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfa40aa-f4c3-4f65-9ab1-75fc3422c3a1 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:35.718132 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfa40aa-f4c3-4f65-9ab1-75fc3422c3a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:35.718270 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfa40aa-f4c3-4f65-9ab1-75fc3422c3a1 admin admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:35.719081 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cfa40aa-f4c3-4f65-9ab1-75fc3422c3a1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:35.722682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cfa40aa-f4c3-4f65-9ab1-75fc3422c3a1 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:35.771030 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1078/2171] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:47:35 2022] POST /identity/v3/users => generated 298 bytes in 286 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:47:35.774851 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9165213c-fdb7-401a-b385-9ed7af80cfd1 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:35.788240 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9165213c-fdb7-401a-b385-9ed7af80cfd1 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:35.795386 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0c600f-8fd8-4750-bb21-ce736af3bb9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:35.796867 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0c600f-8fd8-4750-bb21-ce736af3bb9a admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:35.797303 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0c600f-8fd8-4750-bb21-ce736af3bb9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:35.797913 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0c600f-8fd8-4750-bb21-ce736af3bb9a admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:35.798879 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0c600f-8fd8-4750-bb21-ce736af3bb9a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:35.801994 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0c600f-8fd8-4750-bb21-ce736af3bb9a admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:35.817696 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1094/2172] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:47:35 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:47:35.822069 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cfa40aa-f4c3-4f65-9ab1-75fc3422c3a1 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:35.842883 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cfa40aa-f4c3-4f65-9ab1-75fc3422c3a1 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:35.848584 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb992d5c-bd52-45f1-9ecf-7fad769b00f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:35.849276 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb992d5c-bd52-45f1-9ecf-7fad769b00f5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:35.849574 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb992d5c-bd52-45f1-9ecf-7fad769b00f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:35.849851 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb992d5c-bd52-45f1-9ecf-7fad769b00f5 admin admin] PATH_INFO: `/v3/projects/69046d579b514acfbe4405442cd50b6f/users/0377a1d07dad45369927f80737dbf7fb/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:35.868835 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb992d5c-bd52-45f1-9ecf-7fad769b00f5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=69046d579b514acfbe4405442cd50b6f, user_id=0377a1d07dad45369927f80737dbf7fb, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:35.871534 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb992d5c-bd52-45f1-9ecf-7fad769b00f5 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:35.889757 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1079/2173] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:47:35 2022] PUT /identity/v3/projects/69046d579b514acfbe4405442cd50b6f/users/0377a1d07dad45369927f80737dbf7fb/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:47:35.891164 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0c600f-8fd8-4750-bb21-ce736af3bb9a admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:35.904217 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0c600f-8fd8-4750-bb21-ce736af3bb9a admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:35.937179 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:47:35.978475 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31d0f87-473f-42b3-b26b-f810850b780b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:35.979120 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31d0f87-473f-42b3-b26b-f810850b780b admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:35.979405 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31d0f87-473f-42b3-b26b-f810850b780b admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:35.979687 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31d0f87-473f-42b3-b26b-f810850b780b admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:35.980337 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e31d0f87-473f-42b3-b26b-f810850b780b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:35.981935 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e31d0f87-473f-42b3-b26b-f810850b780b admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:35.995990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1095/2174] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:47:35 2022] GET /identity/v3/roles => generated 1420 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:47:35.997313 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb992d5c-bd52-45f1-9ecf-7fad769b00f5 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:36.008287 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb992d5c-bd52-45f1-9ecf-7fad769b00f5 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:36.013033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e0a4a2-7c41-439f-ab4a-30180436c772 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:36.013794 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e0a4a2-7c41-439f-ab4a-30180436c772 admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:36.014139 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e0a4a2-7c41-439f-ab4a-30180436c772 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:36.014695 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e0a4a2-7c41-439f-ab4a-30180436c772 admin admin] PATH_INFO: `/v3/projects/69046d579b514acfbe4405442cd50b6f/users/0377a1d07dad45369927f80737dbf7fb/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:36.017044 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e0a4a2-7c41-439f-ab4a-30180436c772 admin admin] RBAC: Authorizing `identity:create_grant(project_id=69046d579b514acfbe4405442cd50b6f, user_id=0377a1d07dad45369927f80737dbf7fb, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:36.019842 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e0a4a2-7c41-439f-ab4a-30180436c772 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:36.042317 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1080/2175] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:47:35 2022] PUT /identity/v3/projects/69046d579b514acfbe4405442cd50b6f/users/0377a1d07dad45369927f80737dbf7fb/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:47:36.046626 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e419db-2114-4dd1-8b8f-d4547788bbbd None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:36.047102 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e419db-2114-4dd1-8b8f-d4547788bbbd None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:36.051247 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e419db-2114-4dd1-8b8f-d4547788bbbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:36.093540 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-50e419db-2114-4dd1-8b8f-d4547788bbbd None None] MFA Rules not processed for user `0377a1d07dad45369927f80737dbf7fb`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:47:36.095369 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-50e419db-2114-4dd1-8b8f-d4547788bbbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:36.185226 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1096/2176] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:47:36 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:47:36.200193 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e0a4a2-7c41-439f-ab4a-30180436c772 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:36.202545 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-e7e0a4a2-7c41-439f-ab4a-30180436c772 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:36.275134 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e0a4a2-7c41-439f-ab4a-30180436c772 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:36.280777 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc358816-9e90-451c-af2d-026b214f255f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:36.281405 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc358816-9e90-451c-af2d-026b214f255f service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:36.281685 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc358816-9e90-451c-af2d-026b214f255f service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:36.281967 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc358816-9e90-451c-af2d-026b214f255f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:36.344993 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc358816-9e90-451c-af2d-026b214f255f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:36.346483 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc358816-9e90-451c-af2d-026b214f255f service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:36.357250 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1081/2177] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:47:36 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:47:37.907102 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e419db-2114-4dd1-8b8f-d4547788bbbd None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:37.918775 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e419db-2114-4dd1-8b8f-d4547788bbbd None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:37.922547 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de404f58-17ba-4f8c-9e76-5042977e60a8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:37.923155 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de404f58-17ba-4f8c-9e76-5042977e60a8 service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:37.923467 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de404f58-17ba-4f8c-9e76-5042977e60a8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:37.923739 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de404f58-17ba-4f8c-9e76-5042977e60a8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:37.987741 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de404f58-17ba-4f8c-9e76-5042977e60a8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:37.989180 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de404f58-17ba-4f8c-9e76-5042977e60a8 service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:37.998846 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1097/2178] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:47:37 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:47:38.614886 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc358816-9e90-451c-af2d-026b214f255f service cinder] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:38.623456 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc358816-9e90-451c-af2d-026b214f255f service cinder] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:38.627184 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-999c065f-7010-445f-8522-52df5c24a5b3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:38.627866 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999c065f-7010-445f-8522-52df5c24a5b3 service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:38.628085 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999c065f-7010-445f-8522-52df5c24a5b3 service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:38.628403 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999c065f-7010-445f-8522-52df5c24a5b3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:38.638480 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-999c065f-7010-445f-8522-52df5c24a5b3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:38.640089 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-999c065f-7010-445f-8522-52df5c24a5b3 service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:38.650194 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1082/2179] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:47:38 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:47:42.593160 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de404f58-17ba-4f8c-9e76-5042977e60a8 service cinder] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:42.652431 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:47:42.668159 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de404f58-17ba-4f8c-9e76-5042977e60a8 service cinder] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:42.673743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10beaf0b-9709-4334-b4cd-db4cb2289f41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:42.674650 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10beaf0b-9709-4334-b4cd-db4cb2289f41 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:42.674954 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10beaf0b-9709-4334-b4cd-db4cb2289f41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:42.675259 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10beaf0b-9709-4334-b4cd-db4cb2289f41 admin admin] PATH_INFO: `/v3/users/2dc9e8a3fb4c4c79ad66f4a035e5931b` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:42.676409 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10beaf0b-9709-4334-b4cd-db4cb2289f41 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2dc9e8a3fb4c4c79ad66f4a035e5931b)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:42.678333 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10beaf0b-9709-4334-b4cd-db4cb2289f41 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:42.735722 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-10beaf0b-9709-4334-b4cd-db4cb2289f41 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2dc9e8a3fb4c4c79ad66f4a035e5931b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:42.744861 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-10beaf0b-9709-4334-b4cd-db4cb2289f41 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2dc9e8a3fb4c4c79ad66f4a035e5931b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:42.745480 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-10beaf0b-9709-4334-b4cd-db4cb2289f41 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2dc9e8a3fb4c4c79ad66f4a035e5931b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:42.746087 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-10beaf0b-9709-4334-b4cd-db4cb2289f41 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2dc9e8a3fb4c4c79ad66f4a035e5931b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:42.759975 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-10beaf0b-9709-4334-b4cd-db4cb2289f41 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2dc9e8a3fb4c4c79ad66f4a035e5931b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:42.801588 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1098/2180] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:47:42 2022] DELETE /identity/v3/users/2dc9e8a3fb4c4c79ad66f4a035e5931b => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:47:42.810846 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb802ce2-f1d4-4088-9d04-b98ffb676c80 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:42.811285 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb802ce2-f1d4-4088-9d04-b98ffb676c80 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:42.811396 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb802ce2-f1d4-4088-9d04-b98ffb676c80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:42.840503 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-cb802ce2-f1d4-4088-9d04-b98ffb676c80 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:47:42.841902 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-cb802ce2-f1d4-4088-9d04-b98ffb676c80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:42.919465 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:47:42.963764 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1083/2181] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:47:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:47:42.990577 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10beaf0b-9709-4334-b4cd-db4cb2289f41 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:42.992470 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-10beaf0b-9709-4334-b4cd-db4cb2289f41 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:43.118159 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10beaf0b-9709-4334-b4cd-db4cb2289f41 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:43.127285 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25013b11-20d4-4fe2-ae19-e904d8a03887 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:43.127909 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25013b11-20d4-4fe2-ae19-e904d8a03887 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:43.128195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25013b11-20d4-4fe2-ae19-e904d8a03887 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:43.128572 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25013b11-20d4-4fe2-ae19-e904d8a03887 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:43.150205 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25013b11-20d4-4fe2-ae19-e904d8a03887 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:43.155849 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25013b11-20d4-4fe2-ae19-e904d8a03887 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:43.182794 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1099/2182] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:47:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:47:43.591088 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb802ce2-f1d4-4088-9d04-b98ffb676c80 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:43.592564 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-cb802ce2-f1d4-4088-9d04-b98ffb676c80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:43.604824 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb802ce2-f1d4-4088-9d04-b98ffb676c80 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:43.609597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54446cc7-df7f-436b-8842-cfcbc467d1e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:43.610247 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54446cc7-df7f-436b-8842-cfcbc467d1e1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:43.610577 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54446cc7-df7f-436b-8842-cfcbc467d1e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:43.610856 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54446cc7-df7f-436b-8842-cfcbc467d1e1 admin admin] PATH_INFO: `/v3/projects/585c44f8af1d43ed8d12a98e766fa7e9` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:43.612351 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54446cc7-df7f-436b-8842-cfcbc467d1e1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=585c44f8af1d43ed8d12a98e766fa7e9)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:43.614353 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54446cc7-df7f-436b-8842-cfcbc467d1e1 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:43.669366 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-54446cc7-df7f-436b-8842-cfcbc467d1e1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '585c44f8af1d43ed8d12a98e766fa7e9', 'request_id': 'req-54446cc7-df7f-436b-8842-cfcbc467d1e1'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:43.676633 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-54446cc7-df7f-436b-8842-cfcbc467d1e1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '585c44f8af1d43ed8d12a98e766fa7e9', 'request_id': 'req-54446cc7-df7f-436b-8842-cfcbc467d1e1'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:43.684472 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-54446cc7-df7f-436b-8842-cfcbc467d1e1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '585c44f8af1d43ed8d12a98e766fa7e9', 'request_id': 'req-54446cc7-df7f-436b-8842-cfcbc467d1e1'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:43.696921 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-54446cc7-df7f-436b-8842-cfcbc467d1e1 admin admin] The token cache is being invalidate because project 585c44f8af1d43ed8d12a98e766fa7e9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:47:43.697214 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-54446cc7-df7f-436b-8842-cfcbc467d1e1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:43.697881 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-54446cc7-df7f-436b-8842-cfcbc467d1e1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:43.699493 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1084/2183] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:47:43 2022] DELETE /identity/v3/projects/585c44f8af1d43ed8d12a98e766fa7e9 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:47:43.703573 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25013b11-20d4-4fe2-ae19-e904d8a03887 service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:43.713696 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25013b11-20d4-4fe2-ae19-e904d8a03887 service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:43.782422 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933eebbc-3b45-4b3c-8b26-55b7de2717e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:43.783078 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933eebbc-3b45-4b3c-8b26-55b7de2717e2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:43.783360 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933eebbc-3b45-4b3c-8b26-55b7de2717e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:43.783639 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933eebbc-3b45-4b3c-8b26-55b7de2717e2 admin admin] PATH_INFO: `/v3/users/6bac7605bd0043d69c496418c9a0d17e` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:43.785345 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-933eebbc-3b45-4b3c-8b26-55b7de2717e2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6bac7605bd0043d69c496418c9a0d17e)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:43.787287 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-933eebbc-3b45-4b3c-8b26-55b7de2717e2 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:43.804475 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:47:43.845366 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-933eebbc-3b45-4b3c-8b26-55b7de2717e2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6bac7605bd0043d69c496418c9a0d17e'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:43.853502 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-933eebbc-3b45-4b3c-8b26-55b7de2717e2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6bac7605bd0043d69c496418c9a0d17e'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:43.854409 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-933eebbc-3b45-4b3c-8b26-55b7de2717e2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6bac7605bd0043d69c496418c9a0d17e'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:43.855047 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-933eebbc-3b45-4b3c-8b26-55b7de2717e2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6bac7605bd0043d69c496418c9a0d17e'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:43.862533 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-933eebbc-3b45-4b3c-8b26-55b7de2717e2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6bac7605bd0043d69c496418c9a0d17e'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:43.882907 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1100/2184] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:47:43 2022] DELETE /identity/v3/users/6bac7605bd0043d69c496418c9a0d17e => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:47:44.084675 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535ad27b-1b24-4d7a-8bbb-5ae40bc9ea97 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:44.085246 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535ad27b-1b24-4d7a-8bbb-5ae40bc9ea97 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:44.085246 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535ad27b-1b24-4d7a-8bbb-5ae40bc9ea97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:44.112064 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-535ad27b-1b24-4d7a-8bbb-5ae40bc9ea97 None None] MFA Rules not processed for user `e65d298cca0f478eba502edf5bd13954`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:47:44.114516 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-535ad27b-1b24-4d7a-8bbb-5ae40bc9ea97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:44.250304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1085/2185] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:47:44 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:47:44.369096 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933eebbc-3b45-4b3c-8b26-55b7de2717e2 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:44.370727 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-933eebbc-3b45-4b3c-8b26-55b7de2717e2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:44.465044 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933eebbc-3b45-4b3c-8b26-55b7de2717e2 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:44.471100 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7522721e-d076-4401-ab71-067756e01e48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:44.471749 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7522721e-d076-4401-ab71-067756e01e48 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:44.472566 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7522721e-d076-4401-ab71-067756e01e48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:44.472846 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7522721e-d076-4401-ab71-067756e01e48 admin admin] PATH_INFO: `/v3/projects/c4e99a697e494b02ab34f35d4a505e6d` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:44.474168 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7522721e-d076-4401-ab71-067756e01e48 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c4e99a697e494b02ab34f35d4a505e6d)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:44.476879 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7522721e-d076-4401-ab71-067756e01e48 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:44.604325 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7522721e-d076-4401-ab71-067756e01e48 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c4e99a697e494b02ab34f35d4a505e6d', 'request_id': 'req-7522721e-d076-4401-ab71-067756e01e48'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:44.611335 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7522721e-d076-4401-ab71-067756e01e48 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c4e99a697e494b02ab34f35d4a505e6d', 'request_id': 'req-7522721e-d076-4401-ab71-067756e01e48'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:44.624678 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7522721e-d076-4401-ab71-067756e01e48 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c4e99a697e494b02ab34f35d4a505e6d', 'request_id': 'req-7522721e-d076-4401-ab71-067756e01e48'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:44.641331 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7522721e-d076-4401-ab71-067756e01e48 admin admin] The token cache is being invalidate because project c4e99a697e494b02ab34f35d4a505e6d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:47:44.642547 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7522721e-d076-4401-ab71-067756e01e48 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:44.642547 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7522721e-d076-4401-ab71-067756e01e48 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:44.648847 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1101/2186] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:47:44 2022] DELETE /identity/v3/projects/c4e99a697e494b02ab34f35d4a505e6d => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:47:44.650872 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535ad27b-1b24-4d7a-8bbb-5ae40bc9ea97 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:44.671493 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535ad27b-1b24-4d7a-8bbb-5ae40bc9ea97 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:44.758562 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98db61b5-115d-4921-a7bc-d43ba18c84cb None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:44.759618 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98db61b5-115d-4921-a7bc-d43ba18c84cb None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:44.760303 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98db61b5-115d-4921-a7bc-d43ba18c84cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:44.762778 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5edcc4b2-aaa3-446f-b9d9-10fe8dab4557 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:44.763525 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edcc4b2-aaa3-446f-b9d9-10fe8dab4557 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:44.763872 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edcc4b2-aaa3-446f-b9d9-10fe8dab4557 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:44.764354 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edcc4b2-aaa3-446f-b9d9-10fe8dab4557 admin admin] PATH_INFO: `/v3/users/0377a1d07dad45369927f80737dbf7fb` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:44.765597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5edcc4b2-aaa3-446f-b9d9-10fe8dab4557 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0377a1d07dad45369927f80737dbf7fb)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:44.767892 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5edcc4b2-aaa3-446f-b9d9-10fe8dab4557 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:44.784234 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-98db61b5-115d-4921-a7bc-d43ba18c84cb None None] MFA Rules not processed for user `97c8877c74194a6bbbf25b3c8f5b7dbd`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:47:44.786208 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-98db61b5-115d-4921-a7bc-d43ba18c84cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:44.795713 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:47:44.815029 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:DELETE=16,keystone:SELECT=82,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:47:44.857741 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-5edcc4b2-aaa3-446f-b9d9-10fe8dab4557 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0377a1d07dad45369927f80737dbf7fb'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:44.872640 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-5edcc4b2-aaa3-446f-b9d9-10fe8dab4557 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0377a1d07dad45369927f80737dbf7fb'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:44.873250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1102/2187] 10.222.0.25 () {58 vars in 1009 bytes} [Fri Oct 28 09:47:44 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:47:44.874526 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-5edcc4b2-aaa3-446f-b9d9-10fe8dab4557 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0377a1d07dad45369927f80737dbf7fb'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:44.875194 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-5edcc4b2-aaa3-446f-b9d9-10fe8dab4557 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0377a1d07dad45369927f80737dbf7fb'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:44.884645 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-5edcc4b2-aaa3-446f-b9d9-10fe8dab4557 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0377a1d07dad45369927f80737dbf7fb'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:44.911112 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1086/2188] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:47:44 2022] DELETE /identity/v3/users/0377a1d07dad45369927f80737dbf7fb => generated 0 bytes in 261 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:47:44.934250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98db61b5-115d-4921-a7bc-d43ba18c84cb None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:44.935724 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-98db61b5-115d-4921-a7bc-d43ba18c84cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:45.022071 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98db61b5-115d-4921-a7bc-d43ba18c84cb None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:45.103048 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d45766e-6e95-477b-8dc7-67dc240dcf9a service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c6cffe0d56e45c4b8fb54fbe5df6094', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:45.104124 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d45766e-6e95-477b-8dc7-67dc240dcf9a service swift] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:45.104636 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d45766e-6e95-477b-8dc7-67dc240dcf9a service swift] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:45.105058 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d45766e-6e95-477b-8dc7-67dc240dcf9a service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:45.107767 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-9d45766e-6e95-477b-8dc7-67dc240dcf9a service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:45.192637 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d45766e-6e95-477b-8dc7-67dc240dcf9a service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:45.196270 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d45766e-6e95-477b-8dc7-67dc240dcf9a service swift] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:45.208548 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1103/2189] 10.222.0.25 () {60 vars in 1384 bytes} [Fri Oct 28 09:47:44 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 275 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 28 09:47:45.553214 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5edcc4b2-aaa3-446f-b9d9-10fe8dab4557 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:45.562439 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-5edcc4b2-aaa3-446f-b9d9-10fe8dab4557 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:45.777364 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5edcc4b2-aaa3-446f-b9d9-10fe8dab4557 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:45.782887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e7ef0d1-a161-482f-9332-d98fbac2920d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:45.783744 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7ef0d1-a161-482f-9332-d98fbac2920d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:45.784366 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7ef0d1-a161-482f-9332-d98fbac2920d admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:45.784869 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7ef0d1-a161-482f-9332-d98fbac2920d admin admin] PATH_INFO: `/v3/projects/69046d579b514acfbe4405442cd50b6f` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:45.789198 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e7ef0d1-a161-482f-9332-d98fbac2920d admin admin] RBAC: Authorizing `identity:delete_project(project_id=69046d579b514acfbe4405442cd50b6f)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:45.791849 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e7ef0d1-a161-482f-9332-d98fbac2920d admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:45.875434 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-7e7ef0d1-a161-482f-9332-d98fbac2920d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '69046d579b514acfbe4405442cd50b6f', 'request_id': 'req-7e7ef0d1-a161-482f-9332-d98fbac2920d'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:45.883512 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-7e7ef0d1-a161-482f-9332-d98fbac2920d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '69046d579b514acfbe4405442cd50b6f', 'request_id': 'req-7e7ef0d1-a161-482f-9332-d98fbac2920d'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:45.898126 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-7e7ef0d1-a161-482f-9332-d98fbac2920d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '69046d579b514acfbe4405442cd50b6f', 'request_id': 'req-7e7ef0d1-a161-482f-9332-d98fbac2920d'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:45.917820 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-7e7ef0d1-a161-482f-9332-d98fbac2920d admin admin] The token cache is being invalidate because project 69046d579b514acfbe4405442cd50b6f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:47:45.918228 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-7e7ef0d1-a161-482f-9332-d98fbac2920d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:45.919041 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-7e7ef0d1-a161-482f-9332-d98fbac2920d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:47:45.921409 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1087/2190] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:47:45 2022] DELETE /identity/v3/projects/69046d579b514acfbe4405442cd50b6f => generated 0 bytes in 369 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:47:45.926477 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63eed635-1981-4cf5-9170-9e1bb3004ac6 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:45.927025 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63eed635-1981-4cf5-9170-9e1bb3004ac6 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:45.927025 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63eed635-1981-4cf5-9170-9e1bb3004ac6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:45.998711 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-63eed635-1981-4cf5-9170-9e1bb3004ac6 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:47:45.999987 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-63eed635-1981-4cf5-9170-9e1bb3004ac6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:46.029334 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:47:46.191736 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1104/2191] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:47:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 266 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:47:46.203528 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd82c1bb-9ff4-440d-8d57-d7f8ff5b5eb6 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:46.203528 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd82c1bb-9ff4-440d-8d57-d7f8ff5b5eb6 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:46.203528 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd82c1bb-9ff4-440d-8d57-d7f8ff5b5eb6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:46.253952 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-dd82c1bb-9ff4-440d-8d57-d7f8ff5b5eb6 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:47:46.255398 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-dd82c1bb-9ff4-440d-8d57-d7f8ff5b5eb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:46.291874 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1088/2192] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:47:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:47:46.292415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63eed635-1981-4cf5-9170-9e1bb3004ac6 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:46.323294 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63eed635-1981-4cf5-9170-9e1bb3004ac6 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:46.332638 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eecf971-8540-46ba-9cad-8f7a398f3139 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:46.333679 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eecf971-8540-46ba-9cad-8f7a398f3139 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:46.334183 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eecf971-8540-46ba-9cad-8f7a398f3139 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:46.334710 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eecf971-8540-46ba-9cad-8f7a398f3139 admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:46.335756 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eecf971-8540-46ba-9cad-8f7a398f3139 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:47:46.336317 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eecf971-8540-46ba-9cad-8f7a398f3139 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:46.340819 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eecf971-8540-46ba-9cad-8f7a398f3139 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:46.398165 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1105/2193] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:47:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 120 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:47:46.404146 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd82c1bb-9ff4-440d-8d57-d7f8ff5b5eb6 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:46.429003 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd82c1bb-9ff4-440d-8d57-d7f8ff5b5eb6 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:46.437605 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ae18948-1116-40bd-bb91-1e8e43ec316b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:46.438579 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae18948-1116-40bd-bb91-1e8e43ec316b admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:46.438949 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae18948-1116-40bd-bb91-1e8e43ec316b admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:46.439275 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae18948-1116-40bd-bb91-1e8e43ec316b admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:46.440029 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ae18948-1116-40bd-bb91-1e8e43ec316b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:46.442765 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ae18948-1116-40bd-bb91-1e8e43ec316b admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:46.474072 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:DELETE=15,keystone:SELECT=82,keystone:INSERT=3 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:47:46.493017 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1089/2194] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:47:46 2022] POST /identity/v3/projects => generated 373 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:47:46.499476 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eecf971-8540-46ba-9cad-8f7a398f3139 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:46.514833 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eecf971-8540-46ba-9cad-8f7a398f3139 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:46.618981 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f817f46e-f3fe-4594-a3be-74b8d72e6583 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:46.619746 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f817f46e-f3fe-4594-a3be-74b8d72e6583 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:46.620250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f817f46e-f3fe-4594-a3be-74b8d72e6583 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:46.620606 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f817f46e-f3fe-4594-a3be-74b8d72e6583 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:46.621496 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f817f46e-f3fe-4594-a3be-74b8d72e6583 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:46.623535 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f817f46e-f3fe-4594-a3be-74b8d72e6583 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:46.677900 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1106/2195] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:47:46 2022] POST /identity/v3/users => generated 287 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:47:46.679345 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ae18948-1116-40bd-bb91-1e8e43ec316b admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:46.693850 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ae18948-1116-40bd-bb91-1e8e43ec316b admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:46.699719 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40161e81-0dc5-4460-9001-59281bd2c849 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:46.700640 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40161e81-0dc5-4460-9001-59281bd2c849 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:46.702139 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40161e81-0dc5-4460-9001-59281bd2c849 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:46.702848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40161e81-0dc5-4460-9001-59281bd2c849 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:46.703770 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40161e81-0dc5-4460-9001-59281bd2c849 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:46.706202 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40161e81-0dc5-4460-9001-59281bd2c849 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:46.723710 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1090/2196] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:47:46 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:47:46.726952 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f817f46e-f3fe-4594-a3be-74b8d72e6583 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:46.741828 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f817f46e-f3fe-4594-a3be-74b8d72e6583 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:46.747029 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da5a90b-ca2d-47b2-8019-17afd38ccbab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:46.750562 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da5a90b-ca2d-47b2-8019-17afd38ccbab admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:46.751102 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da5a90b-ca2d-47b2-8019-17afd38ccbab admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:46.754103 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da5a90b-ca2d-47b2-8019-17afd38ccbab admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:46.757297 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0da5a90b-ca2d-47b2-8019-17afd38ccbab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:46.759648 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0da5a90b-ca2d-47b2-8019-17afd38ccbab admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:46.803396 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1107/2197] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:47:46 2022] GET /identity/v3/roles => generated 1420 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:47:46.804918 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40161e81-0dc5-4460-9001-59281bd2c849 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:46.817288 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40161e81-0dc5-4460-9001-59281bd2c849 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:46.832163 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb3cf4a-5923-4312-8501-93280d101e09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:46.834111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb3cf4a-5923-4312-8501-93280d101e09 admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:46.835196 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb3cf4a-5923-4312-8501-93280d101e09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:46.836029 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb3cf4a-5923-4312-8501-93280d101e09 admin admin] PATH_INFO: `/v3/projects/d408e68f0ada4cc5ad76f57c0d1d82f6/users/711c241be21c48509104ab93c38a395b/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:46.868841 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecb3cf4a-5923-4312-8501-93280d101e09 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d408e68f0ada4cc5ad76f57c0d1d82f6, user_id=711c241be21c48509104ab93c38a395b, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:46.871998 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecb3cf4a-5923-4312-8501-93280d101e09 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:46.894797 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1091/2198] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:47:46 2022] PUT /identity/v3/projects/d408e68f0ada4cc5ad76f57c0d1d82f6/users/711c241be21c48509104ab93c38a395b/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 91 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:47:46.970646 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183e67e2-f6f3-4cdf-87cd-2204e476fba4 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:46.971316 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183e67e2-f6f3-4cdf-87cd-2204e476fba4 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:46.971316 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183e67e2-f6f3-4cdf-87cd-2204e476fba4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:47.005935 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-183e67e2-f6f3-4cdf-87cd-2204e476fba4 None None] MFA Rules not processed for user `711c241be21c48509104ab93c38a395b`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:47:47.007264 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-183e67e2-f6f3-4cdf-87cd-2204e476fba4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:47.139109 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1108/2199] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:47:46 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:47:47.153955 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb3cf4a-5923-4312-8501-93280d101e09 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:47:47.155818 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-ecb3cf4a-5923-4312-8501-93280d101e09 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:47:47.273794 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb3cf4a-5923-4312-8501-93280d101e09 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:47:47.285081 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5f7135-e3e1-43fb-9b0f-0e29b8df663f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:47:47.285081 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5f7135-e3e1-43fb-9b0f-0e29b8df663f service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:47:47.285081 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5f7135-e3e1-43fb-9b0f-0e29b8df663f service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:47:47.285081 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5f7135-e3e1-43fb-9b0f-0e29b8df663f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:47:47.297285 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e5f7135-e3e1-43fb-9b0f-0e29b8df663f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:47:47.298959 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e5f7135-e3e1-43fb-9b0f-0e29b8df663f service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:47:47.311015 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1092/2200] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:47:47 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:47:57.133791 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=93,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:47:57.305169 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=42,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:48:00.081146 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183e67e2-f6f3-4cdf-87cd-2204e476fba4 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:48:00.148180 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183e67e2-f6f3-4cdf-87cd-2204e476fba4 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:48:00.153107 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0851b2af-b5df-457d-a95c-180fd9ff174c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:48:00.153820 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0851b2af-b5df-457d-a95c-180fd9ff174c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:48:00.154145 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0851b2af-b5df-457d-a95c-180fd9ff174c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:48:00.154442 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0851b2af-b5df-457d-a95c-180fd9ff174c admin admin] PATH_INFO: `/v3/users/711c241be21c48509104ab93c38a395b` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:48:00.155551 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0851b2af-b5df-457d-a95c-180fd9ff174c admin admin] RBAC: Authorizing `identity:delete_user(user_id=711c241be21c48509104ab93c38a395b)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:48:00.157441 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0851b2af-b5df-457d-a95c-180fd9ff174c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:48:00.208982 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-0851b2af-b5df-457d-a95c-180fd9ff174c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '711c241be21c48509104ab93c38a395b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:48:00.217081 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-0851b2af-b5df-457d-a95c-180fd9ff174c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '711c241be21c48509104ab93c38a395b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:48:00.217701 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-0851b2af-b5df-457d-a95c-180fd9ff174c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '711c241be21c48509104ab93c38a395b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:48:00.218332 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-0851b2af-b5df-457d-a95c-180fd9ff174c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '711c241be21c48509104ab93c38a395b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:48:00.226468 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-0851b2af-b5df-457d-a95c-180fd9ff174c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '711c241be21c48509104ab93c38a395b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:48:00.244277 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1109/2201] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:48:00 2022] DELETE /identity/v3/users/711c241be21c48509104ab93c38a395b => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:48:00.248687 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2516ec-c35e-4a0e-94e1-c20e7bdef1a8 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:48:00.249206 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2516ec-c35e-4a0e-94e1-c20e7bdef1a8 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:48:00.249508 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d2516ec-c35e-4a0e-94e1-c20e7bdef1a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:48:00.267863 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-7d2516ec-c35e-4a0e-94e1-c20e7bdef1a8 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:48:00.269106 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-7d2516ec-c35e-4a0e-94e1-c20e7bdef1a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:48:00.327119 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1093/2202] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:48:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:48:00.337264 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0851b2af-b5df-457d-a95c-180fd9ff174c admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:48:00.338704 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-0851b2af-b5df-457d-a95c-180fd9ff174c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:48:00.396507 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0851b2af-b5df-457d-a95c-180fd9ff174c admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:48:00.400239 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46913789-e303-4003-9e66-23d667e39717 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:48:00.400842 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46913789-e303-4003-9e66-23d667e39717 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:48:00.401313 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46913789-e303-4003-9e66-23d667e39717 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:48:00.401935 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46913789-e303-4003-9e66-23d667e39717 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:48:00.415518 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46913789-e303-4003-9e66-23d667e39717 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:48:00.417051 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46913789-e303-4003-9e66-23d667e39717 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:48:00.428694 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1110/2203] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:48:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:48:00.748148 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2516ec-c35e-4a0e-94e1-c20e7bdef1a8 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:48:00.749547 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-7d2516ec-c35e-4a0e-94e1-c20e7bdef1a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:48:00.762160 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d2516ec-c35e-4a0e-94e1-c20e7bdef1a8 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:48:00.766808 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7def5819-38b8-4922-bad7-87521014e19a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:48:00.767441 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7def5819-38b8-4922-bad7-87521014e19a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:48:00.767726 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7def5819-38b8-4922-bad7-87521014e19a admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:48:00.768003 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7def5819-38b8-4922-bad7-87521014e19a admin admin] PATH_INFO: `/v3/projects/d408e68f0ada4cc5ad76f57c0d1d82f6` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:48:00.769191 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7def5819-38b8-4922-bad7-87521014e19a admin admin] RBAC: Authorizing `identity:delete_project(project_id=d408e68f0ada4cc5ad76f57c0d1d82f6)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:48:00.771179 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7def5819-38b8-4922-bad7-87521014e19a admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:48:00.845448 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-7def5819-38b8-4922-bad7-87521014e19a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd408e68f0ada4cc5ad76f57c0d1d82f6', 'request_id': 'req-7def5819-38b8-4922-bad7-87521014e19a'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:48:00.853692 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-7def5819-38b8-4922-bad7-87521014e19a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd408e68f0ada4cc5ad76f57c0d1d82f6', 'request_id': 'req-7def5819-38b8-4922-bad7-87521014e19a'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:48:00.861741 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-7def5819-38b8-4922-bad7-87521014e19a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd408e68f0ada4cc5ad76f57c0d1d82f6', 'request_id': 'req-7def5819-38b8-4922-bad7-87521014e19a'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:48:00.873945 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-7def5819-38b8-4922-bad7-87521014e19a admin admin] The token cache is being invalidate because project d408e68f0ada4cc5ad76f57c0d1d82f6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:48:00.874234 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-7def5819-38b8-4922-bad7-87521014e19a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:48:00.874917 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-7def5819-38b8-4922-bad7-87521014e19a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:48:00.876400 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1094/2204] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:48:00 2022] DELETE /identity/v3/projects/d408e68f0ada4cc5ad76f57c0d1d82f6 => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:48:05.170165 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f645e06-0bb3-4bf2-8b0b-2cfc3ac19281 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:48:05.170739 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f645e06-0bb3-4bf2-8b0b-2cfc3ac19281 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:48:05.170739 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f645e06-0bb3-4bf2-8b0b-2cfc3ac19281 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:48:05.173085 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1111/2205] 10.222.0.25 () {54 vars in 961 bytes} [Fri Oct 28 09:48:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:48:05.178206 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1016cf77-5dbe-429f-b2b3-5aa91e2e8a77 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:48:05.178759 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1016cf77-5dbe-429f-b2b3-5aa91e2e8a77 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:48:05.178865 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1016cf77-5dbe-429f-b2b3-5aa91e2e8a77 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:48:05.200473 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-1016cf77-5dbe-429f-b2b3-5aa91e2e8a77 None None] MFA Rules not processed for user `3f1d95049d2d4593bf52670b1a0035fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:48:05.201682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-1016cf77-5dbe-429f-b2b3-5aa91e2e8a77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:48:05.267323 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1095/2206] 10.222.0.25 () {58 vars in 1058 bytes} [Fri Oct 28 09:48:05 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:48:05.274961 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f645e06-0bb3-4bf2-8b0b-2cfc3ac19281 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:48:05.291533 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f645e06-0bb3-4bf2-8b0b-2cfc3ac19281 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:48:05.297569 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94682e0-819c-4cb6-b0be-a8ca21fd5a07 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3f1d95049d2d4593bf52670b1a0035fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:48:05.298245 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94682e0-819c-4cb6-b0be-a8ca21fd5a07 service placement] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:48:05.298613 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94682e0-819c-4cb6-b0be-a8ca21fd5a07 service placement] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:48:05.298926 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94682e0-819c-4cb6-b0be-a8ca21fd5a07 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:48:05.300840 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a94682e0-819c-4cb6-b0be-a8ca21fd5a07 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:48:05.312407 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a94682e0-819c-4cb6-b0be-a8ca21fd5a07 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:48:05.314080 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a94682e0-819c-4cb6-b0be-a8ca21fd5a07 service placement] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:48:05.326687 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1112/2207] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:48:05 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:48:06.468711 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc665507-74c0-4f93-9d12-37c00ebd8d7c None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:48:06.469299 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc665507-74c0-4f93-9d12-37c00ebd8d7c None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:48:06.470017 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc665507-74c0-4f93-9d12-37c00ebd8d7c None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:48:06.471488 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1096/2208] 10.222.0.25 () {54 vars in 921 bytes} [Fri Oct 28 09:48:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:48:06.478806 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4966a5a5-0028-4b01-9ff9-0aefe518f23f None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:48:06.479447 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4966a5a5-0028-4b01-9ff9-0aefe518f23f None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:48:06.479447 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4966a5a5-0028-4b01-9ff9-0aefe518f23f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:48:06.504236 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-4966a5a5-0028-4b01-9ff9-0aefe518f23f None None] MFA Rules not processed for user `688ceb201d304c2da5373c3cf36c2d79`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:48:06.505311 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4966a5a5-0028-4b01-9ff9-0aefe518f23f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:48:06.509907 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:48:06.590123 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1113/2209] 10.222.0.25 () {58 vars in 1018 bytes} [Fri Oct 28 09:48:06 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:48:06.604641 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc665507-74c0-4f93-9d12-37c00ebd8d7c None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:48:06.606307 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-fc665507-74c0-4f93-9d12-37c00ebd8d7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:48:06.629184 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:48:06.667328 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc665507-74c0-4f93-9d12-37c00ebd8d7c None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:48:06.673091 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47802899-3a8c-44b8-a274-0c85733c9eaf service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4b2ea9aa31445d69e0c1d96b5657708', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:48:06.673846 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47802899-3a8c-44b8-a274-0c85733c9eaf service glance] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:48:06.674215 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47802899-3a8c-44b8-a274-0c85733c9eaf service glance] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:48:06.674628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47802899-3a8c-44b8-a274-0c85733c9eaf service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:48:06.685316 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47802899-3a8c-44b8-a274-0c85733c9eaf service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:48:06.686702 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47802899-3a8c-44b8-a274-0c85733c9eaf service glance] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:48:06.699102 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1097/2210] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:48:06 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:48:06.772798 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be7593f-696a-4678-bebb-7a179f5c9adb None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:48:06.773087 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be7593f-696a-4678-bebb-7a179f5c9adb None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:48:06.773365 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be7593f-696a-4678-bebb-7a179f5c9adb None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:48:06.774735 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1114/2211] 10.222.0.25 () {54 vars in 921 bytes} [Fri Oct 28 09:48:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:48:06.781207 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807afec9-9f0c-4688-981c-aa05995d7a1b None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:48:06.781491 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807afec9-9f0c-4688-981c-aa05995d7a1b None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:48:06.781767 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807afec9-9f0c-4688-981c-aa05995d7a1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:48:06.802000 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-807afec9-9f0c-4688-981c-aa05995d7a1b None None] MFA Rules not processed for user `1db6b98e87e64584932447c82fefc3b7`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:48:06.803236 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-807afec9-9f0c-4688-981c-aa05995d7a1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:48:06.855466 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1098/2212] 10.222.0.25 () {58 vars in 1018 bytes} [Fri Oct 28 09:48:06 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:48:06.868006 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be7593f-696a-4678-bebb-7a179f5c9adb None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:48:06.869668 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-2be7593f-696a-4678-bebb-7a179f5c9adb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:48:06.882583 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be7593f-696a-4678-bebb-7a179f5c9adb None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:48:06.886344 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a957d206-1802-4ecd-be52-6c0a989c22a1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:48:06.886962 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a957d206-1802-4ecd-be52-6c0a989c22a1 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:48:06.887240 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a957d206-1802-4ecd-be52-6c0a989c22a1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:48:06.887510 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a957d206-1802-4ecd-be52-6c0a989c22a1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:48:06.894488 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a957d206-1802-4ecd-be52-6c0a989c22a1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:48:06.895877 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a957d206-1802-4ecd-be52-6c0a989c22a1 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:48:06.905584 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1115/2213] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:48:06 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:48:06.991521 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d396ad0-c397-4b9c-9694-c3d1d64733e1 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:48:06.991809 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d396ad0-c397-4b9c-9694-c3d1d64733e1 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:48:06.992087 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d396ad0-c397-4b9c-9694-c3d1d64733e1 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:48:06.993429 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1099/2214] 10.222.0.25 () {54 vars in 921 bytes} [Fri Oct 28 09:48:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:48:06.998290 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b9a1e4-1979-4677-947f-675d80a146d5 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:48:06.998676 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b9a1e4-1979-4677-947f-675d80a146d5 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:48:06.998953 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8b9a1e4-1979-4677-947f-675d80a146d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:48:07.018097 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-c8b9a1e4-1979-4677-947f-675d80a146d5 None None] MFA Rules not processed for user `688ceb201d304c2da5373c3cf36c2d79`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:48:07.019318 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-c8b9a1e4-1979-4677-947f-675d80a146d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:48:07.028724 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1116/2215] 10.222.0.25 () {58 vars in 1018 bytes} [Fri Oct 28 09:48:06 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:48:07.040151 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d396ad0-c397-4b9c-9694-c3d1d64733e1 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:48:07.041840 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-6d396ad0-c397-4b9c-9694-c3d1d64733e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:48:07.055843 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d396ad0-c397-4b9c-9694-c3d1d64733e1 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:48:07.059646 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb6c91c-4889-4815-89a9-1538165959a5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:48:07.060246 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb6c91c-4889-4815-89a9-1538165959a5 service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:48:07.060528 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb6c91c-4889-4815-89a9-1538165959a5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:48:07.060823 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb6c91c-4889-4815-89a9-1538165959a5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:48:07.071448 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb6c91c-4889-4815-89a9-1538165959a5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:48:07.072923 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb6c91c-4889-4815-89a9-1538165959a5 service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:48:07.084941 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1100/2216] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:48:07 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:48:17.013571 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=36 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:48:17.079469 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=46 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:48:20.176873 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6c1218-89f6-4e7c-9a68-b3cf5bf40f09 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:48:20.177360 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6c1218-89f6-4e7c-9a68-b3cf5bf40f09 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:48:20.177470 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6c1218-89f6-4e7c-9a68-b3cf5bf40f09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:48:20.198254 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-9b6c1218-89f6-4e7c-9a68-b3cf5bf40f09 None None] MFA Rules not processed for user `97c8877c74194a6bbbf25b3c8f5b7dbd`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:48:20.199506 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-9b6c1218-89f6-4e7c-9a68-b3cf5bf40f09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:48:20.258256 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1117/2217] 10.222.0.25 () {58 vars in 1009 bytes} [Fri Oct 28 09:48:20 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:48:20.270279 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb6c91c-4889-4815-89a9-1538165959a5 service neutron] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:48:20.271744 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-7eb6c91c-4889-4815-89a9-1538165959a5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:48:20.342949 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb6c91c-4889-4815-89a9-1538165959a5 service neutron] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:48:20.348595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dcc93c2-b997-4045-982a-5c423fa5a6cd service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c6cffe0d56e45c4b8fb54fbe5df6094', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:48:20.349202 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dcc93c2-b997-4045-982a-5c423fa5a6cd service swift] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:48:20.349483 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dcc93c2-b997-4045-982a-5c423fa5a6cd service swift] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:48:20.349756 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dcc93c2-b997-4045-982a-5c423fa5a6cd service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:48:20.358593 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dcc93c2-b997-4045-982a-5c423fa5a6cd service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:48:20.360106 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dcc93c2-b997-4045-982a-5c423fa5a6cd service swift] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:48:20.369018 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1101/2218] 10.222.0.25 () {60 vars in 1384 bytes} [Fri Oct 28 09:48:20 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 28 09:48:30.251438 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=25 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:48:30.365557 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=32 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:48:50.593916 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd54c43f-21b4-4ba1-bfd0-d03b85432a2e None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:48:50.594461 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd54c43f-21b4-4ba1-bfd0-d03b85432a2e None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:48:50.594461 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd54c43f-21b4-4ba1-bfd0-d03b85432a2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:48:50.614984 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-fd54c43f-21b4-4ba1-bfd0-d03b85432a2e None None] MFA Rules not processed for user `97c8877c74194a6bbbf25b3c8f5b7dbd`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:48:50.616187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-fd54c43f-21b4-4ba1-bfd0-d03b85432a2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:48:50.623327 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1118/2219] 10.222.0.25 () {58 vars in 967 bytes} [Fri Oct 28 09:48:50 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:48:50.645589 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dcc93c2-b997-4045-982a-5c423fa5a6cd service swift] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:48:50.656052 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dcc93c2-b997-4045-982a-5c423fa5a6cd service swift] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:48:50.661680 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0666b92c-f077-4b55-bfd5-fa1216c819af service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c6cffe0d56e45c4b8fb54fbe5df6094', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:48:50.662284 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0666b92c-f077-4b55-bfd5-fa1216c819af service swift] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:48:50.662624 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0666b92c-f077-4b55-bfd5-fa1216c819af service swift] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:48:50.662901 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0666b92c-f077-4b55-bfd5-fa1216c819af service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:48:50.671255 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0666b92c-f077-4b55-bfd5-fa1216c819af service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:48:50.672829 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0666b92c-f077-4b55-bfd5-fa1216c819af service swift] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:48:50.680893 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1102/2220] 10.222.0.25 () {60 vars in 1384 bytes} [Fri Oct 28 09:48:50 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 28 09:48:51.245507 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289a050b-cebf-4fba-b2cc-b43fa8f27e95 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:48:51.245793 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289a050b-cebf-4fba-b2cc-b43fa8f27e95 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:48:51.246067 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-289a050b-cebf-4fba-b2cc-b43fa8f27e95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:48:51.265360 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-289a050b-cebf-4fba-b2cc-b43fa8f27e95 None None] MFA Rules not processed for user `ec23c6593e7a446298d607294997664c`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:48:51.266616 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-289a050b-cebf-4fba-b2cc-b43fa8f27e95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:48:51.328808 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1119/2221] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:48:51 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:49:00.677632 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=6 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:49:01.320981 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=28 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:49:33.381758 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cabc63e4-43f2-4e7b-8543-ebaa17173e32 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:49:33.382359 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cabc63e4-43f2-4e7b-8543-ebaa17173e32 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:49:33.382359 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cabc63e4-43f2-4e7b-8543-ebaa17173e32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:49:33.401755 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-cabc63e4-43f2-4e7b-8543-ebaa17173e32 None None] MFA Rules not processed for user `e65d298cca0f478eba502edf5bd13954`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:49:33.403014 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-cabc63e4-43f2-4e7b-8543-ebaa17173e32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:49:33.462531 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1103/2222] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:49:33 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:49:38.291096 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2e1562-1967-4a98-b4bc-4345487eb477 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:49:38.291842 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2e1562-1967-4a98-b4bc-4345487eb477 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:49:38.291842 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2e1562-1967-4a98-b4bc-4345487eb477 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:49:38.313193 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-eb2e1562-1967-4a98-b4bc-4345487eb477 None None] MFA Rules not processed for user `ed9a4c24cbfb4689b7f984219c7503a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:49:38.314968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-eb2e1562-1967-4a98-b4bc-4345487eb477 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:49:38.376329 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1120/2223] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:49:38 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:49:43.455699 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=25 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:49:48.366026 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=25 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:49:54.942181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cabc63e4-43f2-4e7b-8543-ebaa17173e32 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:49:54.943667 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-cabc63e4-43f2-4e7b-8543-ebaa17173e32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:49:55.027044 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cabc63e4-43f2-4e7b-8543-ebaa17173e32 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:49:55.031971 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e584a4-8543-4a5d-9d26-38f3756814b5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:49:55.033099 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e584a4-8543-4a5d-9d26-38f3756814b5 service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:49:55.033536 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e584a4-8543-4a5d-9d26-38f3756814b5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:49:55.033993 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e584a4-8543-4a5d-9d26-38f3756814b5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:49:55.036122 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-45e584a4-8543-4a5d-9d26-38f3756814b5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:49:55.065564 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e584a4-8543-4a5d-9d26-38f3756814b5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:49:55.067397 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e584a4-8543-4a5d-9d26-38f3756814b5 service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:49:55.087549 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1104/2224] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:49:54 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:49:56.836615 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2e1562-1967-4a98-b4bc-4345487eb477 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:49:56.846763 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2e1562-1967-4a98-b4bc-4345487eb477 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:49:56.850435 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d04030-c12f-4e2c-bcd0-a3231d27320e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:49:56.851258 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d04030-c12f-4e2c-bcd0-a3231d27320e service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:49:56.851539 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d04030-c12f-4e2c-bcd0-a3231d27320e service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:49:56.851810 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d04030-c12f-4e2c-bcd0-a3231d27320e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:49:56.853539 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-19d04030-c12f-4e2c-bcd0-a3231d27320e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:49:56.867845 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d04030-c12f-4e2c-bcd0-a3231d27320e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:49:56.869289 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d04030-c12f-4e2c-bcd0-a3231d27320e service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:49:56.881111 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1121/2225] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:49:56 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:50:03.911063 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98cc75b-b874-4d70-97d5-e46dd437f7eb None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:03.911905 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98cc75b-b874-4d70-97d5-e46dd437f7eb None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:03.912048 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98cc75b-b874-4d70-97d5-e46dd437f7eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:03.947513 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-d98cc75b-b874-4d70-97d5-e46dd437f7eb None None] MFA Rules not processed for user `97c8877c74194a6bbbf25b3c8f5b7dbd`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:50:03.953258 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-d98cc75b-b874-4d70-97d5-e46dd437f7eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:50:03.974387 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1105/2226] 10.222.0.25 () {58 vars in 967 bytes} [Fri Oct 28 09:50:03 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 65 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:50:04.077826 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d04030-c12f-4e2c-bcd0-a3231d27320e service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:50:04.095331 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d04030-c12f-4e2c-bcd0-a3231d27320e service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:50:04.103622 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d66499-fdab-44ea-af17-12002bb6d320 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c6cffe0d56e45c4b8fb54fbe5df6094', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:50:04.104646 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d66499-fdab-44ea-af17-12002bb6d320 service swift] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:04.105093 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d66499-fdab-44ea-af17-12002bb6d320 service swift] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:04.105529 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d66499-fdab-44ea-af17-12002bb6d320 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:04.118906 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d66499-fdab-44ea-af17-12002bb6d320 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:50:04.121967 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d66499-fdab-44ea-af17-12002bb6d320 service swift] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:50:04.135552 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1122/2227] 10.222.0.25 () {60 vars in 1384 bytes} [Fri Oct 28 09:50:04 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 58 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 28 09:50:04.448428 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98cc75b-b874-4d70-97d5-e46dd437f7eb None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:50:04.450978 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-d98cc75b-b874-4d70-97d5-e46dd437f7eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:50:04.619744 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98cc75b-b874-4d70-97d5-e46dd437f7eb None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:50:04.626062 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a7629f3-bdc9-4599-b3dc-ff79b5ea47df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:50:04.627043 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7629f3-bdc9-4599-b3dc-ff79b5ea47df admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:04.627669 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7629f3-bdc9-4599-b3dc-ff79b5ea47df admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:04.628896 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7629f3-bdc9-4599-b3dc-ff79b5ea47df admin admin] PATH_INFO: `/v3/users/ec23c6593e7a446298d607294997664c` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:04.630279 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a7629f3-bdc9-4599-b3dc-ff79b5ea47df admin admin] RBAC: Authorizing `identity:delete_user(user_id=ec23c6593e7a446298d607294997664c)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:50:04.632326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a7629f3-bdc9-4599-b3dc-ff79b5ea47df admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:50:04.701415 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-2a7629f3-bdc9-4599-b3dc-ff79b5ea47df admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ec23c6593e7a446298d607294997664c'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:50:04.716638 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-2a7629f3-bdc9-4599-b3dc-ff79b5ea47df admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ec23c6593e7a446298d607294997664c'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:50:04.718706 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-2a7629f3-bdc9-4599-b3dc-ff79b5ea47df admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ec23c6593e7a446298d607294997664c'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:50:04.721621 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-2a7629f3-bdc9-4599-b3dc-ff79b5ea47df admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ec23c6593e7a446298d607294997664c'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:50:04.732386 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-2a7629f3-bdc9-4599-b3dc-ff79b5ea47df admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ec23c6593e7a446298d607294997664c'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:50:04.757269 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1106/2228] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:50:04 2022] DELETE /identity/v3/users/ec23c6593e7a446298d607294997664c => generated 0 bytes in 310 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:50:05.395515 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7741978e-4b66-4059-be24-ee1f735f3277 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:05.396261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7741978e-4b66-4059-be24-ee1f735f3277 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:05.396261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7741978e-4b66-4059-be24-ee1f735f3277 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:05.420465 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-7741978e-4b66-4059-be24-ee1f735f3277 None None] MFA Rules not processed for user `715a782e5a56431e88e4fe5a3b0a62aa`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:50:05.421907 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7741978e-4b66-4059-be24-ee1f735f3277 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:50:05.502278 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1123/2229] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:50:05 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:50:05.874194 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a7629f3-bdc9-4599-b3dc-ff79b5ea47df admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:50:05.875959 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-2a7629f3-bdc9-4599-b3dc-ff79b5ea47df admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:50:05.899398 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:50:05.967809 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a7629f3-bdc9-4599-b3dc-ff79b5ea47df admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:50:05.975889 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca92544-e75d-4f9b-a1b4-3ea856df7456 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:50:05.975889 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca92544-e75d-4f9b-a1b4-3ea856df7456 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:05.976687 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca92544-e75d-4f9b-a1b4-3ea856df7456 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:05.977083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca92544-e75d-4f9b-a1b4-3ea856df7456 admin admin] PATH_INFO: `/v3/projects/9a20b51b10ce463da8f9ad280063bb09` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:05.978807 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca92544-e75d-4f9b-a1b4-3ea856df7456 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9a20b51b10ce463da8f9ad280063bb09)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:50:05.981049 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca92544-e75d-4f9b-a1b4-3ea856df7456 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:50:06.061437 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-cca92544-e75d-4f9b-a1b4-3ea856df7456 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9a20b51b10ce463da8f9ad280063bb09', 'request_id': 'req-cca92544-e75d-4f9b-a1b4-3ea856df7456'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:50:06.069699 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-cca92544-e75d-4f9b-a1b4-3ea856df7456 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9a20b51b10ce463da8f9ad280063bb09', 'request_id': 'req-cca92544-e75d-4f9b-a1b4-3ea856df7456'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:50:06.082501 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-cca92544-e75d-4f9b-a1b4-3ea856df7456 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9a20b51b10ce463da8f9ad280063bb09', 'request_id': 'req-cca92544-e75d-4f9b-a1b4-3ea856df7456'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:50:06.098502 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-cca92544-e75d-4f9b-a1b4-3ea856df7456 admin admin] The token cache is being invalidate because project 9a20b51b10ce463da8f9ad280063bb09 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:50:06.098822 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-cca92544-e75d-4f9b-a1b4-3ea856df7456 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:50:06.099637 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-cca92544-e75d-4f9b-a1b4-3ea856df7456 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:50:06.102991 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1107/2230] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:50:05 2022] DELETE /identity/v3/projects/9a20b51b10ce463da8f9ad280063bb09 => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:50:08.709538 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7741978e-4b66-4059-be24-ee1f735f3277 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:50:08.712274 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7741978e-4b66-4059-be24-ee1f735f3277 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:50:08.817426 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7741978e-4b66-4059-be24-ee1f735f3277 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:50:08.920298 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba4c7802-0072-4875-8803-ca0b376063e3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:50:08.920942 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4c7802-0072-4875-8803-ca0b376063e3 service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:08.921225 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4c7802-0072-4875-8803-ca0b376063e3 service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:08.921585 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4c7802-0072-4875-8803-ca0b376063e3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:08.925487 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ba4c7802-0072-4875-8803-ca0b376063e3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:50:08.954317 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:50:09.040051 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba4c7802-0072-4875-8803-ca0b376063e3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:50:09.041954 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba4c7802-0072-4875-8803-ca0b376063e3 service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:50:09.055573 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1124/2231] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:50:08 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:50:11.250488 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ee6916-42b5-4c48-adab-38babdbf1407 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:11.251130 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ee6916-42b5-4c48-adab-38babdbf1407 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:11.252333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ee6916-42b5-4c48-adab-38babdbf1407 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:11.256780 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1108/2232] 10.222.0.25 () {54 vars in 921 bytes} [Fri Oct 28 09:50:11 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:50:11.308068 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4072508a-b0f0-4801-9534-6ec83ab3ab66 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:11.308712 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4072508a-b0f0-4801-9534-6ec83ab3ab66 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:11.308712 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4072508a-b0f0-4801-9534-6ec83ab3ab66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:11.332470 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-4072508a-b0f0-4801-9534-6ec83ab3ab66 None None] MFA Rules not processed for user `3f1d95049d2d4593bf52670b1a0035fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:50:11.333806 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4072508a-b0f0-4801-9534-6ec83ab3ab66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:50:11.452134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1125/2233] 10.222.0.25 () {58 vars in 1018 bytes} [Fri Oct 28 09:50:11 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:50:11.471570 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ee6916-42b5-4c48-adab-38babdbf1407 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:50:11.474104 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-03ee6916-42b5-4c48-adab-38babdbf1407 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:50:11.496141 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ee6916-42b5-4c48-adab-38babdbf1407 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:50:11.506032 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f71d6f-ea75-4e9e-bc58-b78f4a8acf92 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3f1d95049d2d4593bf52670b1a0035fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:50:11.507167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f71d6f-ea75-4e9e-bc58-b78f4a8acf92 service placement] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:11.507664 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f71d6f-ea75-4e9e-bc58-b78f4a8acf92 service placement] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:11.508167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f71d6f-ea75-4e9e-bc58-b78f4a8acf92 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:11.526725 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35f71d6f-ea75-4e9e-bc58-b78f4a8acf92 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:50:11.530079 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35f71d6f-ea75-4e9e-bc58-b78f4a8acf92 service placement] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:50:11.550758 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1109/2234] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:50:11 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:50:17.231728 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4072508a-b0f0-4801-9534-6ec83ab3ab66 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:50:17.233925 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4072508a-b0f0-4801-9534-6ec83ab3ab66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:50:17.329813 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4072508a-b0f0-4801-9534-6ec83ab3ab66 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:50:17.334791 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09f32d28-6636-4e1d-aac5-9d9b32a26926 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:50:17.335608 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f32d28-6636-4e1d-aac5-9d9b32a26926 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:17.335990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f32d28-6636-4e1d-aac5-9d9b32a26926 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:17.336357 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f32d28-6636-4e1d-aac5-9d9b32a26926 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:17.338614 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-09f32d28-6636-4e1d-aac5-9d9b32a26926 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:50:17.435833 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09f32d28-6636-4e1d-aac5-9d9b32a26926 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:50:17.437505 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09f32d28-6636-4e1d-aac5-9d9b32a26926 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:50:17.450039 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1126/2235] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:50:17 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 220 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:50:17.746036 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f71d6f-ea75-4e9e-bc58-b78f4a8acf92 service placement] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:50:17.752443 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-35f71d6f-ea75-4e9e-bc58-b78f4a8acf92 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:50:17.834384 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f71d6f-ea75-4e9e-bc58-b78f4a8acf92 service placement] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:50:17.839934 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb1b47c2-d117-43d1-88ab-8a25536ef7b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:50:17.840755 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1b47c2-d117-43d1-88ab-8a25536ef7b5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:17.841043 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1b47c2-d117-43d1-88ab-8a25536ef7b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:17.841348 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1b47c2-d117-43d1-88ab-8a25536ef7b5 admin admin] PATH_INFO: `/v3/users/23acf07cede847a69ca839ee200accd3` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:17.843352 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb1b47c2-d117-43d1-88ab-8a25536ef7b5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=23acf07cede847a69ca839ee200accd3)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:50:17.845480 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb1b47c2-d117-43d1-88ab-8a25536ef7b5 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:50:17.901043 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:50:17.916906 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-bb1b47c2-d117-43d1-88ab-8a25536ef7b5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '23acf07cede847a69ca839ee200accd3'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:50:17.929378 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-bb1b47c2-d117-43d1-88ab-8a25536ef7b5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '23acf07cede847a69ca839ee200accd3'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:50:17.930167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-bb1b47c2-d117-43d1-88ab-8a25536ef7b5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '23acf07cede847a69ca839ee200accd3'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:50:17.931012 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-bb1b47c2-d117-43d1-88ab-8a25536ef7b5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '23acf07cede847a69ca839ee200accd3'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:50:17.951190 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-bb1b47c2-d117-43d1-88ab-8a25536ef7b5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '23acf07cede847a69ca839ee200accd3'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:50:17.993776 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1110/2236] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:50:17 2022] DELETE /identity/v3/users/23acf07cede847a69ca839ee200accd3 => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:50:18.367064 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09f32d28-6636-4e1d-aac5-9d9b32a26926 service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:50:18.378788 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:50:18.389604 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09f32d28-6636-4e1d-aac5-9d9b32a26926 service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:50:18.501603 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b472f5e8-54b7-4ccd-b31a-7be1375b939a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:50:18.502671 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b472f5e8-54b7-4ccd-b31a-7be1375b939a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:18.503140 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b472f5e8-54b7-4ccd-b31a-7be1375b939a admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:18.503619 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b472f5e8-54b7-4ccd-b31a-7be1375b939a admin admin] PATH_INFO: `/v3/projects/6f99f17a6cab46039419eb3b44bdd142` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:18.505724 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b472f5e8-54b7-4ccd-b31a-7be1375b939a admin admin] RBAC: Authorizing `identity:delete_project(project_id=6f99f17a6cab46039419eb3b44bdd142)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:50:18.509214 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b472f5e8-54b7-4ccd-b31a-7be1375b939a admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:50:18.601672 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-b472f5e8-54b7-4ccd-b31a-7be1375b939a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6f99f17a6cab46039419eb3b44bdd142', 'request_id': 'req-b472f5e8-54b7-4ccd-b31a-7be1375b939a'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:50:18.610907 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-b472f5e8-54b7-4ccd-b31a-7be1375b939a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6f99f17a6cab46039419eb3b44bdd142', 'request_id': 'req-b472f5e8-54b7-4ccd-b31a-7be1375b939a'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:50:18.621709 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-b472f5e8-54b7-4ccd-b31a-7be1375b939a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6f99f17a6cab46039419eb3b44bdd142', 'request_id': 'req-b472f5e8-54b7-4ccd-b31a-7be1375b939a'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:50:18.637729 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-b472f5e8-54b7-4ccd-b31a-7be1375b939a admin admin] The token cache is being invalidate because project 6f99f17a6cab46039419eb3b44bdd142 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:50:18.638038 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-b472f5e8-54b7-4ccd-b31a-7be1375b939a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:50:18.640284 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-b472f5e8-54b7-4ccd-b31a-7be1375b939a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:50:18.641347 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1127/2237] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:50:18 2022] DELETE /identity/v3/projects/6f99f17a6cab46039419eb3b44bdd142 => generated 0 bytes in 275 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:50:18.646281 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb1b47c2-d117-43d1-88ab-8a25536ef7b5 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:50:18.649572 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-bb1b47c2-d117-43d1-88ab-8a25536ef7b5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:50:18.677758 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb1b47c2-d117-43d1-88ab-8a25536ef7b5 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:50:18.786126 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44058910-4f65-4bfe-bc20-01be986067f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:50:18.787348 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44058910-4f65-4bfe-bc20-01be986067f1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:18.787860 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44058910-4f65-4bfe-bc20-01be986067f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:18.788239 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44058910-4f65-4bfe-bc20-01be986067f1 admin admin] PATH_INFO: `/v3/users/eb94858904f14e0aa1d31a2a5f6c48b1` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:18.790735 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44058910-4f65-4bfe-bc20-01be986067f1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=eb94858904f14e0aa1d31a2a5f6c48b1)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:50:18.792888 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44058910-4f65-4bfe-bc20-01be986067f1 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:50:18.855086 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-44058910-4f65-4bfe-bc20-01be986067f1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'eb94858904f14e0aa1d31a2a5f6c48b1'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:50:18.864853 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-44058910-4f65-4bfe-bc20-01be986067f1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'eb94858904f14e0aa1d31a2a5f6c48b1'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:50:18.865986 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-44058910-4f65-4bfe-bc20-01be986067f1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'eb94858904f14e0aa1d31a2a5f6c48b1'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:50:18.867228 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-44058910-4f65-4bfe-bc20-01be986067f1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'eb94858904f14e0aa1d31a2a5f6c48b1'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:50:18.881409 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-44058910-4f65-4bfe-bc20-01be986067f1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'eb94858904f14e0aa1d31a2a5f6c48b1'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:50:18.918547 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1111/2238] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:50:18 2022] DELETE /identity/v3/users/eb94858904f14e0aa1d31a2a5f6c48b1 => generated 0 bytes in 273 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:50:19.221609 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b472f5e8-54b7-4ccd-b31a-7be1375b939a admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:50:19.224076 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-b472f5e8-54b7-4ccd-b31a-7be1375b939a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:50:19.266307 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b472f5e8-54b7-4ccd-b31a-7be1375b939a admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:50:19.362608 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e664d4ce-86b5-42bd-aa08-1e394eeac374 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:50:19.363357 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e664d4ce-86b5-42bd-aa08-1e394eeac374 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:19.363648 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e664d4ce-86b5-42bd-aa08-1e394eeac374 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:19.363921 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e664d4ce-86b5-42bd-aa08-1e394eeac374 admin admin] PATH_INFO: `/v3/projects/211973c63e014864a20f470121b94a0f` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:19.365287 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e664d4ce-86b5-42bd-aa08-1e394eeac374 admin admin] RBAC: Authorizing `identity:delete_project(project_id=211973c63e014864a20f470121b94a0f)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:50:19.367316 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e664d4ce-86b5-42bd-aa08-1e394eeac374 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:50:19.435349 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:50:19.450615 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e664d4ce-86b5-42bd-aa08-1e394eeac374 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '211973c63e014864a20f470121b94a0f', 'request_id': 'req-e664d4ce-86b5-42bd-aa08-1e394eeac374'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:50:19.458481 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e664d4ce-86b5-42bd-aa08-1e394eeac374 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '211973c63e014864a20f470121b94a0f', 'request_id': 'req-e664d4ce-86b5-42bd-aa08-1e394eeac374'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:50:19.468578 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e664d4ce-86b5-42bd-aa08-1e394eeac374 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '211973c63e014864a20f470121b94a0f', 'request_id': 'req-e664d4ce-86b5-42bd-aa08-1e394eeac374'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:50:19.484268 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e664d4ce-86b5-42bd-aa08-1e394eeac374 admin admin] The token cache is being invalidate because project 211973c63e014864a20f470121b94a0f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:50:19.484750 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e664d4ce-86b5-42bd-aa08-1e394eeac374 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:50:19.485599 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e664d4ce-86b5-42bd-aa08-1e394eeac374 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:50:19.487641 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1128/2239] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:50:19 2022] DELETE /identity/v3/projects/211973c63e014864a20f470121b94a0f => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:50:19.493891 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca815c8f-df88-4adf-ae4e-a5915aea644f None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:19.494589 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca815c8f-df88-4adf-ae4e-a5915aea644f None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:19.495040 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca815c8f-df88-4adf-ae4e-a5915aea644f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:19.528953 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-ca815c8f-df88-4adf-ae4e-a5915aea644f None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:50:19.530269 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-ca815c8f-df88-4adf-ae4e-a5915aea644f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:50:19.588801 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:50:19.648543 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1112/2240] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:50:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:50:19.665682 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fdb0a31-b54d-4466-8a33-6a67d6af4d2f None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:19.666165 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fdb0a31-b54d-4466-8a33-6a67d6af4d2f None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:19.666584 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fdb0a31-b54d-4466-8a33-6a67d6af4d2f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:19.692441 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-9fdb0a31-b54d-4466-8a33-6a67d6af4d2f None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:50:19.693781 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-9fdb0a31-b54d-4466-8a33-6a67d6af4d2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:50:19.705655 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1129/2241] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:50:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:50:19.710123 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca815c8f-df88-4adf-ae4e-a5915aea644f None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:50:19.722270 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca815c8f-df88-4adf-ae4e-a5915aea644f None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:50:19.736909 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c868092-6f6b-4e6d-be6e-4bf2574b96cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:50:19.737494 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c868092-6f6b-4e6d-be6e-4bf2574b96cd admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:19.737891 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c868092-6f6b-4e6d-be6e-4bf2574b96cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:19.738223 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c868092-6f6b-4e6d-be6e-4bf2574b96cd admin admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:19.739099 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c868092-6f6b-4e6d-be6e-4bf2574b96cd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:50:19.739431 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c868092-6f6b-4e6d-be6e-4bf2574b96cd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:50:19.741302 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c868092-6f6b-4e6d-be6e-4bf2574b96cd admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:50:19.764061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1113/2242] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:50:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:50:19.765859 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fdb0a31-b54d-4466-8a33-6a67d6af4d2f None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:50:19.780165 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fdb0a31-b54d-4466-8a33-6a67d6af4d2f None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:50:19.788045 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a7d21d-58dd-48de-9549-0e3782af1262 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:50:19.789079 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a7d21d-58dd-48de-9549-0e3782af1262 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:19.789592 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a7d21d-58dd-48de-9549-0e3782af1262 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:19.790130 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a7d21d-58dd-48de-9549-0e3782af1262 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:19.791365 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a7d21d-58dd-48de-9549-0e3782af1262 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:50:19.794792 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a7d21d-58dd-48de-9549-0e3782af1262 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:50:19.848822 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1130/2243] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:50:19 2022] POST /identity/v3/projects => generated 379 bytes in 83 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:50:19.855968 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c868092-6f6b-4e6d-be6e-4bf2574b96cd admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:50:19.876377 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c868092-6f6b-4e6d-be6e-4bf2574b96cd admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:50:20.044751 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e1d4f3-9414-4a3c-bd72-e7af79e1e9c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:50:20.046094 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e1d4f3-9414-4a3c-bd72-e7af79e1e9c8 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:20.046990 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e1d4f3-9414-4a3c-bd72-e7af79e1e9c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:20.047682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e1d4f3-9414-4a3c-bd72-e7af79e1e9c8 admin admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:20.049529 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e1d4f3-9414-4a3c-bd72-e7af79e1e9c8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:50:20.055029 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e1d4f3-9414-4a3c-bd72-e7af79e1e9c8 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:50:20.111614 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1114/2244] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:50:19 2022] POST /identity/v3/users => generated 290 bytes in 257 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:50:20.117452 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a7d21d-58dd-48de-9549-0e3782af1262 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:50:20.140786 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a7d21d-58dd-48de-9549-0e3782af1262 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:50:20.152463 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-593c665c-68fe-4ca8-aeea-32ac8af4bfc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:50:20.153613 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593c665c-68fe-4ca8-aeea-32ac8af4bfc5 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:20.154127 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593c665c-68fe-4ca8-aeea-32ac8af4bfc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:20.154685 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-593c665c-68fe-4ca8-aeea-32ac8af4bfc5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:20.155960 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-593c665c-68fe-4ca8-aeea-32ac8af4bfc5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:50:20.159582 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-593c665c-68fe-4ca8-aeea-32ac8af4bfc5 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:50:20.185587 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1131/2245] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:50:20 2022] GET /identity/v3/roles => generated 1420 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:50:20.188524 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e1d4f3-9414-4a3c-bd72-e7af79e1e9c8 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:50:20.215390 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e1d4f3-9414-4a3c-bd72-e7af79e1e9c8 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:50:20.228393 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40145f03-e679-4e17-bbe9-52252aa7a775 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:50:20.230894 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40145f03-e679-4e17-bbe9-52252aa7a775 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:20.231854 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40145f03-e679-4e17-bbe9-52252aa7a775 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:20.232633 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40145f03-e679-4e17-bbe9-52252aa7a775 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:20.234005 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40145f03-e679-4e17-bbe9-52252aa7a775 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:50:20.237986 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40145f03-e679-4e17-bbe9-52252aa7a775 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:50:20.263174 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1115/2246] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:50:20 2022] GET /identity/v3/roles => generated 1420 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:50:20.268429 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-593c665c-68fe-4ca8-aeea-32ac8af4bfc5 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:50:20.287704 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-593c665c-68fe-4ca8-aeea-32ac8af4bfc5 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:50:20.294130 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e15bb7a8-a656-4fbd-b585-afd4fcefe02f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:50:20.295253 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15bb7a8-a656-4fbd-b585-afd4fcefe02f admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:20.295774 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15bb7a8-a656-4fbd-b585-afd4fcefe02f admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:20.296154 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15bb7a8-a656-4fbd-b585-afd4fcefe02f admin admin] PATH_INFO: `/v3/projects/6d071a4f24b541d4b5d49017707e0788/users/a0e517bf245e44b99fd23325307ba2cb/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:20.319471 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e15bb7a8-a656-4fbd-b585-afd4fcefe02f admin admin] RBAC: Authorizing `identity:create_grant(project_id=6d071a4f24b541d4b5d49017707e0788, user_id=a0e517bf245e44b99fd23325307ba2cb, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:50:20.322759 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e15bb7a8-a656-4fbd-b585-afd4fcefe02f admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:50:20.341505 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1132/2247] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:50:20 2022] PUT /identity/v3/projects/6d071a4f24b541d4b5d49017707e0788/users/a0e517bf245e44b99fd23325307ba2cb/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 74 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:50:20.347901 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b17d2c-0d19-48c2-96db-1de8abc98ae0 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:20.348256 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b17d2c-0d19-48c2-96db-1de8abc98ae0 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:20.348568 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b17d2c-0d19-48c2-96db-1de8abc98ae0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:20.374809 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-79b17d2c-0d19-48c2-96db-1de8abc98ae0 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:50:20.376399 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-79b17d2c-0d19-48c2-96db-1de8abc98ae0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:50:20.471807 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1116/2248] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:50:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:50:20.487476 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e15bb7a8-a656-4fbd-b585-afd4fcefe02f admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:50:20.490429 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e15bb7a8-a656-4fbd-b585-afd4fcefe02f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:50:20.597530 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e15bb7a8-a656-4fbd-b585-afd4fcefe02f admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:50:20.603917 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c49118ad-837d-40e3-8401-1180fecc9514 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:50:20.605136 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49118ad-837d-40e3-8401-1180fecc9514 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:20.605469 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49118ad-837d-40e3-8401-1180fecc9514 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:20.605926 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c49118ad-837d-40e3-8401-1180fecc9514 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:20.619963 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c49118ad-837d-40e3-8401-1180fecc9514 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:50:20.622348 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c49118ad-837d-40e3-8401-1180fecc9514 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:50:20.638947 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1133/2249] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:50:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:50:26.049369 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b17d2c-0d19-48c2-96db-1de8abc98ae0 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:50:26.052296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-79b17d2c-0d19-48c2-96db-1de8abc98ae0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:50:26.178772 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:50:26.211211 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b17d2c-0d19-48c2-96db-1de8abc98ae0 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:50:26.218442 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-769b5b66-0c7d-4202-9e00-cf0923fb17ad service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:50:26.218773 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769b5b66-0c7d-4202-9e00-cf0923fb17ad service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:26.220109 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769b5b66-0c7d-4202-9e00-cf0923fb17ad service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:26.220109 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769b5b66-0c7d-4202-9e00-cf0923fb17ad service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:26.222999 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-769b5b66-0c7d-4202-9e00-cf0923fb17ad service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:50:26.404777 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-769b5b66-0c7d-4202-9e00-cf0923fb17ad service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:50:26.413974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-769b5b66-0c7d-4202-9e00-cf0923fb17ad service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:50:26.459827 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1117/2250] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:50:26 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 412 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:50:27.934108 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a9671a-57c0-4bd9-b9a0-578f0e72975e None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:27.934688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a9671a-57c0-4bd9-b9a0-578f0e72975e None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:27.935128 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a9671a-57c0-4bd9-b9a0-578f0e72975e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:27.963853 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-63a9671a-57c0-4bd9-b9a0-578f0e72975e None None] MFA Rules not processed for user `a4237ca76acd4f7483aaf66e1f5d2eea`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:50:27.965812 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-63a9671a-57c0-4bd9-b9a0-578f0e72975e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:50:28.126523 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1134/2251] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:50:27 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:50:28.351035 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6d4661-8e41-429a-b29d-1c4a3f6ca485 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:28.351640 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6d4661-8e41-429a-b29d-1c4a3f6ca485 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:28.352028 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6d4661-8e41-429a-b29d-1c4a3f6ca485 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:28.416548 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-bc6d4661-8e41-429a-b29d-1c4a3f6ca485 None None] MFA Rules not processed for user `a0e517bf245e44b99fd23325307ba2cb`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:50:28.419066 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-bc6d4661-8e41-429a-b29d-1c4a3f6ca485 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:50:28.535303 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1118/2252] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:50:28 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:50:28.541583 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a9671a-57c0-4bd9-b9a0-578f0e72975e None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:50:28.565884 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a9671a-57c0-4bd9-b9a0-578f0e72975e None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:50:28.572107 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b199dcec-9f10-40b7-85ce-00ffd79ca0cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:50:28.572751 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b199dcec-9f10-40b7-85ce-00ffd79ca0cf admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:28.573035 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b199dcec-9f10-40b7-85ce-00ffd79ca0cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:28.573351 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b199dcec-9f10-40b7-85ce-00ffd79ca0cf admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:28.574010 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b199dcec-9f10-40b7-85ce-00ffd79ca0cf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:50:28.576159 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b199dcec-9f10-40b7-85ce-00ffd79ca0cf admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:50:28.585763 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=93,keystone:DELETE=4,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:50:28.619656 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1135/2253] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:50:28 2022] POST /identity/v3/projects => generated 377 bytes in 78 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:50:28.623636 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6d4661-8e41-429a-b29d-1c4a3f6ca485 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:50:28.638972 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6d4661-8e41-429a-b29d-1c4a3f6ca485 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:50:28.759825 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e796fda3-2134-4e57-a255-bef152cc1403 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:50:28.759825 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e796fda3-2134-4e57-a255-bef152cc1403 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:28.759825 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e796fda3-2134-4e57-a255-bef152cc1403 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:28.760375 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e796fda3-2134-4e57-a255-bef152cc1403 admin admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:28.760782 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e796fda3-2134-4e57-a255-bef152cc1403 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:50:28.762803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e796fda3-2134-4e57-a255-bef152cc1403 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:50:28.794920 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1119/2254] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:50:28 2022] POST /identity/v3/users => generated 295 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:50:28.799048 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b199dcec-9f10-40b7-85ce-00ffd79ca0cf admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:50:28.818135 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e796fda3-2134-4e57-a255-bef152cc1403 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:50:28.824337 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b199dcec-9f10-40b7-85ce-00ffd79ca0cf admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:50:28.830908 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1db2aa-1721-48a8-9af3-8b5f5d229c8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:50:28.833497 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1db2aa-1721-48a8-9af3-8b5f5d229c8f admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:28.833786 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1db2aa-1721-48a8-9af3-8b5f5d229c8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:28.834071 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1db2aa-1721-48a8-9af3-8b5f5d229c8f admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:28.834807 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec1db2aa-1721-48a8-9af3-8b5f5d229c8f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:50:28.836466 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec1db2aa-1721-48a8-9af3-8b5f5d229c8f admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:50:28.845683 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:50:28.848303 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1136/2255] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:50:28 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:50:28.853800 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1db2aa-1721-48a8-9af3-8b5f5d229c8f admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:50:28.869024 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1db2aa-1721-48a8-9af3-8b5f5d229c8f admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:50:28.884366 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f16422-753d-4ddc-a86f-85bdf5f2f822 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:50:28.885526 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f16422-753d-4ddc-a86f-85bdf5f2f822 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:28.885823 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f16422-753d-4ddc-a86f-85bdf5f2f822 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:28.886094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f16422-753d-4ddc-a86f-85bdf5f2f822 admin admin] PATH_INFO: `/v3/projects/baf7f0be1a84414fb5e39b4d164db5f3/users/e48195948fb6421fbecd9bfd8aebe5da/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:28.911707 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25f16422-753d-4ddc-a86f-85bdf5f2f822 admin admin] RBAC: Authorizing `identity:create_grant(project_id=baf7f0be1a84414fb5e39b4d164db5f3, user_id=e48195948fb6421fbecd9bfd8aebe5da, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:50:28.914494 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25f16422-753d-4ddc-a86f-85bdf5f2f822 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:50:28.953846 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1137/2256] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:50:28 2022] PUT /identity/v3/projects/baf7f0be1a84414fb5e39b4d164db5f3/users/e48195948fb6421fbecd9bfd8aebe5da/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:50:28.958052 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f16422-753d-4ddc-a86f-85bdf5f2f822 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:50:28.968581 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e796fda3-2134-4e57-a255-bef152cc1403 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:50:29.095983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-098b3300-a81e-46ce-9b1a-93bf1921ea12 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:50:29.095983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098b3300-a81e-46ce-9b1a-93bf1921ea12 service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:29.097805 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098b3300-a81e-46ce-9b1a-93bf1921ea12 service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:29.098340 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098b3300-a81e-46ce-9b1a-93bf1921ea12 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:29.102287 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-098b3300-a81e-46ce-9b1a-93bf1921ea12 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:50:29.119078 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f16422-753d-4ddc-a86f-85bdf5f2f822 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:50:29.131614 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c47a87-232c-4dd0-aebb-b7e553445f78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:50:29.134535 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c47a87-232c-4dd0-aebb-b7e553445f78 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:29.135205 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c47a87-232c-4dd0-aebb-b7e553445f78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:29.136714 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c47a87-232c-4dd0-aebb-b7e553445f78 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:29.137769 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c47a87-232c-4dd0-aebb-b7e553445f78 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:50:29.140581 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c47a87-232c-4dd0-aebb-b7e553445f78 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:50:29.141298 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-098b3300-a81e-46ce-9b1a-93bf1921ea12 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:50:29.144378 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-098b3300-a81e-46ce-9b1a-93bf1921ea12 service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:50:29.155092 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1138/2257] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:50:28 2022] GET /identity/v3/roles => generated 1420 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:50:29.157442 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1120/2258] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:50:28 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 340 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:50:29.160770 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c47a87-232c-4dd0-aebb-b7e553445f78 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:50:29.190584 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c47a87-232c-4dd0-aebb-b7e553445f78 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:50:29.202790 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42bb8479-157e-46c3-8404-b61b168d3702 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:50:29.203573 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42bb8479-157e-46c3-8404-b61b168d3702 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:29.203841 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42bb8479-157e-46c3-8404-b61b168d3702 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:29.204448 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42bb8479-157e-46c3-8404-b61b168d3702 admin admin] PATH_INFO: `/v3/projects/baf7f0be1a84414fb5e39b4d164db5f3/users/e48195948fb6421fbecd9bfd8aebe5da/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:29.212104 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42bb8479-157e-46c3-8404-b61b168d3702 admin admin] RBAC: Authorizing `identity:create_grant(project_id=baf7f0be1a84414fb5e39b4d164db5f3, user_id=e48195948fb6421fbecd9bfd8aebe5da, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:50:29.215808 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42bb8479-157e-46c3-8404-b61b168d3702 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:50:29.230441 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1139/2259] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:50:29 2022] PUT /identity/v3/projects/baf7f0be1a84414fb5e39b4d164db5f3/users/e48195948fb6421fbecd9bfd8aebe5da/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 70 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:50:29.914767 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-098b3300-a81e-46ce-9b1a-93bf1921ea12 service neutron] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:50:29.926240 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-098b3300-a81e-46ce-9b1a-93bf1921ea12 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:50:30.065365 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-098b3300-a81e-46ce-9b1a-93bf1921ea12 service neutron] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:50:30.252281 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c2bec0-2b8f-4a37-a8be-76828e0af901 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3f1d95049d2d4593bf52670b1a0035fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:50:30.253571 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c2bec0-2b8f-4a37-a8be-76828e0af901 service placement] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:30.254599 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c2bec0-2b8f-4a37-a8be-76828e0af901 service placement] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:30.255000 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c2bec0-2b8f-4a37-a8be-76828e0af901 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:30.257769 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-e9c2bec0-2b8f-4a37-a8be-76828e0af901 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:50:30.277877 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9c2bec0-2b8f-4a37-a8be-76828e0af901 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:50:30.280135 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9c2bec0-2b8f-4a37-a8be-76828e0af901 service placement] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:50:30.293233 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1121/2260] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:50:29 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:50:36.117362 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42bb8479-157e-46c3-8404-b61b168d3702 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:50:36.120400 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-42bb8479-157e-46c3-8404-b61b168d3702 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:50:36.589918 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42bb8479-157e-46c3-8404-b61b168d3702 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:50:36.640143 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b47f118-d990-4dc0-8d1b-3148c2f000c8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:50:36.641339 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b47f118-d990-4dc0-8d1b-3148c2f000c8 service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:36.641883 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b47f118-d990-4dc0-8d1b-3148c2f000c8 service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:36.650969 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b47f118-d990-4dc0-8d1b-3148c2f000c8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:36.653895 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-8b47f118-d990-4dc0-8d1b-3148c2f000c8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:50:36.685117 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b47f118-d990-4dc0-8d1b-3148c2f000c8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:50:36.687057 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b47f118-d990-4dc0-8d1b-3148c2f000c8 service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:50:36.716677 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1140/2261] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:50:36 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 600 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:50:36.933562 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d008b32b-73d3-4d2f-a7bc-617ca12c303a None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:36.934998 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d008b32b-73d3-4d2f-a7bc-617ca12c303a None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:36.935482 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d008b32b-73d3-4d2f-a7bc-617ca12c303a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:36.969925 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:50:37.106747 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b47f118-d990-4dc0-8d1b-3148c2f000c8 service nova] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:50:37.122013 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-d008b32b-73d3-4d2f-a7bc-617ca12c303a None None] MFA Rules not processed for user `e48195948fb6421fbecd9bfd8aebe5da`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:50:37.125362 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-d008b32b-73d3-4d2f-a7bc-617ca12c303a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:50:37.148532 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:50:37.258671 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b47f118-d990-4dc0-8d1b-3148c2f000c8 service nova] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:50:37.271583 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f82689-dbee-4221-8b18-b6a5a462a71d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:50:37.272723 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f82689-dbee-4221-8b18-b6a5a462a71d service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:37.273217 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f82689-dbee-4221-8b18-b6a5a462a71d service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:37.273696 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f82689-dbee-4221-8b18-b6a5a462a71d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:37.276022 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-24f82689-dbee-4221-8b18-b6a5a462a71d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:50:37.300694 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f82689-dbee-4221-8b18-b6a5a462a71d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:50:37.302599 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f82689-dbee-4221-8b18-b6a5a462a71d service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:50:37.327824 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1141/2262] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:50:37 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 221 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:50:37.446476 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1122/2263] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:50:36 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 509 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:50:37.470675 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f82689-dbee-4221-8b18-b6a5a462a71d service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:50:37.519180 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f82689-dbee-4221-8b18-b6a5a462a71d service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:50:37.530541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22edbc25-96e4-448f-9338-a6771a640d70 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:50:37.532608 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22edbc25-96e4-448f-9338-a6771a640d70 service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:37.533195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22edbc25-96e4-448f-9338-a6771a640d70 service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:37.534030 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22edbc25-96e4-448f-9338-a6771a640d70 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:37.736587 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22edbc25-96e4-448f-9338-a6771a640d70 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:50:37.739721 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22edbc25-96e4-448f-9338-a6771a640d70 service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:50:37.785592 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1142/2264] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:50:37 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:50:46.472471 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d008b32b-73d3-4d2f-a7bc-617ca12c303a None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:50:46.487040 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d008b32b-73d3-4d2f-a7bc-617ca12c303a None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:50:46.493749 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c4a151-410b-47d0-b3b4-764c1c447c2e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:50:46.494524 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c4a151-410b-47d0-b3b4-764c1c447c2e service nova] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:46.494879 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c4a151-410b-47d0-b3b4-764c1c447c2e service nova] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:46.495455 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c4a151-410b-47d0-b3b4-764c1c447c2e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:46.497649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-21c4a151-410b-47d0-b3b4-764c1c447c2e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:50:46.511003 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c4a151-410b-47d0-b3b4-764c1c447c2e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:50:46.512662 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c4a151-410b-47d0-b3b4-764c1c447c2e service nova] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:50:46.524443 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1123/2265] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:50:46 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:50:47.767636 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=45 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:50:48.840293 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22edbc25-96e4-448f-9338-a6771a640d70 service nova] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:50:48.840293 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-22edbc25-96e4-448f-9338-a6771a640d70 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:50:48.860933 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22edbc25-96e4-448f-9338-a6771a640d70 service nova] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:50:48.868694 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a229aeb-4c19-4ffd-a7ea-165b13bff718 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:50:48.869382 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a229aeb-4c19-4ffd-a7ea-165b13bff718 service nova] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:48.869706 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a229aeb-4c19-4ffd-a7ea-165b13bff718 service nova] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:48.870021 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a229aeb-4c19-4ffd-a7ea-165b13bff718 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:48.872255 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-1a229aeb-4c19-4ffd-a7ea-165b13bff718 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:50:48.960779 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a229aeb-4c19-4ffd-a7ea-165b13bff718 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:50:48.963007 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a229aeb-4c19-4ffd-a7ea-165b13bff718 service nova] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:50:48.973914 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1143/2266] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:50:48 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:50:54.051590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c4a151-410b-47d0-b3b4-764c1c447c2e service nova] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:50:54.064237 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c4a151-410b-47d0-b3b4-764c1c447c2e service nova] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:50:54.070163 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae89de80-55ae-495e-9372-902902138bee service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:50:54.070909 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae89de80-55ae-495e-9372-902902138bee service nova] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:54.071187 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae89de80-55ae-495e-9372-902902138bee service nova] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:54.071459 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae89de80-55ae-495e-9372-902902138bee service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:54.073286 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-ae89de80-55ae-495e-9372-902902138bee service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:50:54.087138 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae89de80-55ae-495e-9372-902902138bee service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:50:54.088780 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae89de80-55ae-495e-9372-902902138bee service nova] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:50:54.102482 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1124/2267] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:50:54 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:50:54.563351 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a229aeb-4c19-4ffd-a7ea-165b13bff718 service nova] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:50:54.565715 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-1a229aeb-4c19-4ffd-a7ea-165b13bff718 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:50:54.586227 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a229aeb-4c19-4ffd-a7ea-165b13bff718 service nova] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:50:54.590650 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c378d4c9-3151-4909-b006-2d1269ee0965 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:50:54.591509 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c378d4c9-3151-4909-b006-2d1269ee0965 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:54.592024 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c378d4c9-3151-4909-b006-2d1269ee0965 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:54.592530 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c378d4c9-3151-4909-b006-2d1269ee0965 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:54.595038 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-c378d4c9-3151-4909-b006-2d1269ee0965 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:50:54.705403 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c378d4c9-3151-4909-b006-2d1269ee0965 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:50:54.707575 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c378d4c9-3151-4909-b006-2d1269ee0965 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:50:54.722186 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1144/2268] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:50:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:50:58.717657 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae89de80-55ae-495e-9372-902902138bee service nova] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:50:58.719948 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-ae89de80-55ae-495e-9372-902902138bee service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:50:58.748765 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae89de80-55ae-495e-9372-902902138bee service nova] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:50:58.758003 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b864687-e64f-42fe-8f77-0c8af2617447 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:50:58.759050 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b864687-e64f-42fe-8f77-0c8af2617447 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:58.759337 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b864687-e64f-42fe-8f77-0c8af2617447 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:58.759620 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b864687-e64f-42fe-8f77-0c8af2617447 admin admin] PATH_INFO: `/v3/users/ed9a4c24cbfb4689b7f984219c7503a4` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:58.761397 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b864687-e64f-42fe-8f77-0c8af2617447 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ed9a4c24cbfb4689b7f984219c7503a4)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:50:58.763529 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b864687-e64f-42fe-8f77-0c8af2617447 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:50:58.866363 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-2b864687-e64f-42fe-8f77-0c8af2617447 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ed9a4c24cbfb4689b7f984219c7503a4'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:50:58.880956 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-2b864687-e64f-42fe-8f77-0c8af2617447 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ed9a4c24cbfb4689b7f984219c7503a4'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:50:58.881856 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-2b864687-e64f-42fe-8f77-0c8af2617447 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ed9a4c24cbfb4689b7f984219c7503a4'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:50:58.882582 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-2b864687-e64f-42fe-8f77-0c8af2617447 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ed9a4c24cbfb4689b7f984219c7503a4'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:50:58.947571 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-2b864687-e64f-42fe-8f77-0c8af2617447 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ed9a4c24cbfb4689b7f984219c7503a4'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:50:58.969963 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf39d64-2122-428a-bf39-15cfda44d1f9 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:58.971053 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf39d64-2122-428a-bf39-15cfda44d1f9 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:58.971570 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf39d64-2122-428a-bf39-15cfda44d1f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:58.991434 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1125/2269] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:50:58 2022] DELETE /identity/v3/users/ed9a4c24cbfb4689b7f984219c7503a4 => generated 0 bytes in 274 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:50:59.007897 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-2bf39d64-2122-428a-bf39-15cfda44d1f9 None None] MFA Rules not processed for user `a0e517bf245e44b99fd23325307ba2cb`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:50:59.009323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-2bf39d64-2122-428a-bf39-15cfda44d1f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:50:59.115215 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1145/2270] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:50:58 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:50:59.236312 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b864687-e64f-42fe-8f77-0c8af2617447 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:50:59.237872 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-2b864687-e64f-42fe-8f77-0c8af2617447 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:50:59.290066 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:50:59.337064 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b864687-e64f-42fe-8f77-0c8af2617447 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:50:59.342737 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de5b69c-7608-4ddf-8563-2b32150181e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:50:59.343412 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de5b69c-7608-4ddf-8563-2b32150181e0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:59.343730 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de5b69c-7608-4ddf-8563-2b32150181e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:59.344032 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de5b69c-7608-4ddf-8563-2b32150181e0 admin admin] PATH_INFO: `/v3/projects/227ac6c4e642467ca3b9e0bf7181d5c9` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:59.345294 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de5b69c-7608-4ddf-8563-2b32150181e0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=227ac6c4e642467ca3b9e0bf7181d5c9)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:50:59.347331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de5b69c-7608-4ddf-8563-2b32150181e0 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:50:59.411765 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-8de5b69c-7608-4ddf-8563-2b32150181e0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '227ac6c4e642467ca3b9e0bf7181d5c9', 'request_id': 'req-8de5b69c-7608-4ddf-8563-2b32150181e0'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:50:59.421142 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-8de5b69c-7608-4ddf-8563-2b32150181e0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '227ac6c4e642467ca3b9e0bf7181d5c9', 'request_id': 'req-8de5b69c-7608-4ddf-8563-2b32150181e0'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:50:59.432054 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-8de5b69c-7608-4ddf-8563-2b32150181e0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '227ac6c4e642467ca3b9e0bf7181d5c9', 'request_id': 'req-8de5b69c-7608-4ddf-8563-2b32150181e0'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:50:59.451929 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-8de5b69c-7608-4ddf-8563-2b32150181e0 admin admin] The token cache is being invalidate because project 227ac6c4e642467ca3b9e0bf7181d5c9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:50:59.452464 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-8de5b69c-7608-4ddf-8563-2b32150181e0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:50:59.454098 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-8de5b69c-7608-4ddf-8563-2b32150181e0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:50:59.456852 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1126/2271] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:50:59 2022] DELETE /identity/v3/projects/227ac6c4e642467ca3b9e0bf7181d5c9 => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:50:59.462385 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0617be0-8cc4-4b60-b645-17d375ef16b7 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:59.462704 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0617be0-8cc4-4b60-b645-17d375ef16b7 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:59.463004 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0617be0-8cc4-4b60-b645-17d375ef16b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:59.486121 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-f0617be0-8cc4-4b60-b645-17d375ef16b7 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:50:59.492556 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-f0617be0-8cc4-4b60-b645-17d375ef16b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:50:59.530161 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:50:59.560248 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1146/2272] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:50:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:50:59.568893 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e937bad-f4e8-458e-8ada-63b71d2cdc03 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:59.569175 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e937bad-f4e8-458e-8ada-63b71d2cdc03 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:59.569454 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e937bad-f4e8-458e-8ada-63b71d2cdc03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:59.591260 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-1e937bad-f4e8-458e-8ada-63b71d2cdc03 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:50:59.592528 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-1e937bad-f4e8-458e-8ada-63b71d2cdc03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:50:59.600978 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1127/2273] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:50:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:50:59.605452 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0617be0-8cc4-4b60-b645-17d375ef16b7 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:50:59.616749 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0617be0-8cc4-4b60-b645-17d375ef16b7 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:50:59.621994 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f4368d-79a2-4a04-a233-1cb918f9622a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:50:59.622699 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f4368d-79a2-4a04-a233-1cb918f9622a admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:59.622982 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f4368d-79a2-4a04-a233-1cb918f9622a admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:59.623478 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f4368d-79a2-4a04-a233-1cb918f9622a admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:59.623938 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95f4368d-79a2-4a04-a233-1cb918f9622a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:50:59.624249 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95f4368d-79a2-4a04-a233-1cb918f9622a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:50:59.626516 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95f4368d-79a2-4a04-a233-1cb918f9622a admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:50:59.640762 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1147/2274] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:50:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:50:59.645212 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e937bad-f4e8-458e-8ada-63b71d2cdc03 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:50:59.655294 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e937bad-f4e8-458e-8ada-63b71d2cdc03 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:50:59.662331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87ab6b79-3c23-429d-9765-624b3aa66bfc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:50:59.662924 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ab6b79-3c23-429d-9765-624b3aa66bfc admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:59.663208 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ab6b79-3c23-429d-9765-624b3aa66bfc admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:59.663481 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ab6b79-3c23-429d-9765-624b3aa66bfc admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:59.664149 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87ab6b79-3c23-429d-9765-624b3aa66bfc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:50:59.666162 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87ab6b79-3c23-429d-9765-624b3aa66bfc admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:50:59.701700 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1128/2275] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:50:59 2022] POST /identity/v3/projects => generated 373 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:50:59.708459 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f4368d-79a2-4a04-a233-1cb918f9622a admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:50:59.722927 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f4368d-79a2-4a04-a233-1cb918f9622a admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:50:59.807296 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab30e26-0e09-47da-aa12-c2a96882474b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:50:59.808326 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab30e26-0e09-47da-aa12-c2a96882474b admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:59.809256 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab30e26-0e09-47da-aa12-c2a96882474b admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:59.810249 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab30e26-0e09-47da-aa12-c2a96882474b admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:59.811680 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ab30e26-0e09-47da-aa12-c2a96882474b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:50:59.815659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ab30e26-0e09-47da-aa12-c2a96882474b admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:50:59.856608 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1148/2276] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:50:59 2022] POST /identity/v3/users => generated 287 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:50:59.860491 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87ab6b79-3c23-429d-9765-624b3aa66bfc admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:50:59.879829 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87ab6b79-3c23-429d-9765-624b3aa66bfc admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:50:59.888762 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f30b84-1e33-401d-b0f5-f884d0c1830d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:50:59.889981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f30b84-1e33-401d-b0f5-f884d0c1830d admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:59.890710 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f30b84-1e33-401d-b0f5-f884d0c1830d admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:59.891312 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f30b84-1e33-401d-b0f5-f884d0c1830d admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:59.892641 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f30b84-1e33-401d-b0f5-f884d0c1830d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:50:59.896010 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f30b84-1e33-401d-b0f5-f884d0c1830d admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:50:59.916351 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1129/2277] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:50:59 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:50:59.921318 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab30e26-0e09-47da-aa12-c2a96882474b admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:50:59.937827 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab30e26-0e09-47da-aa12-c2a96882474b admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:50:59.946817 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b36987-c651-4415-a5ec-8cba3029deac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:50:59.947656 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b36987-c651-4415-a5ec-8cba3029deac admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:50:59.948113 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b36987-c651-4415-a5ec-8cba3029deac admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:50:59.948566 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b36987-c651-4415-a5ec-8cba3029deac admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:50:59.949775 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b36987-c651-4415-a5ec-8cba3029deac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:50:59.952585 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b36987-c651-4415-a5ec-8cba3029deac admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:50:59.976396 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f30b84-1e33-401d-b0f5-f884d0c1830d admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:50:59.978587 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1149/2278] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:50:59 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:50:59.997419 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f30b84-1e33-401d-b0f5-f884d0c1830d admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:00.004966 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70dbe29-003b-4b81-9c0b-a19e20813d23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:00.005789 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70dbe29-003b-4b81-9c0b-a19e20813d23 admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:00.006184 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70dbe29-003b-4b81-9c0b-a19e20813d23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:00.006607 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70dbe29-003b-4b81-9c0b-a19e20813d23 admin admin] PATH_INFO: `/v3/projects/25e4cd851ba947ed96286977236953a6/users/838463b8eca244218e50fa343d4b5545/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:00.033308 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70dbe29-003b-4b81-9c0b-a19e20813d23 admin admin] RBAC: Authorizing `identity:create_grant(project_id=25e4cd851ba947ed96286977236953a6, user_id=838463b8eca244218e50fa343d4b5545, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:00.036026 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70dbe29-003b-4b81-9c0b-a19e20813d23 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:00.057432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1130/2279] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:50:59 2022] PUT /identity/v3/projects/25e4cd851ba947ed96286977236953a6/users/838463b8eca244218e50fa343d4b5545/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 82 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:51:00.072207 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd62ff43-9b5b-4fdc-868d-2f39d449e9ad None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:00.072579 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd62ff43-9b5b-4fdc-868d-2f39d449e9ad None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:00.072917 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd62ff43-9b5b-4fdc-868d-2f39d449e9ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:00.111004 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-dd62ff43-9b5b-4fdc-868d-2f39d449e9ad None None] MFA Rules not processed for user `838463b8eca244218e50fa343d4b5545`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:51:00.112729 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-dd62ff43-9b5b-4fdc-868d-2f39d449e9ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:00.203481 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1150/2280] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:51:00 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:51:00.358673 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70dbe29-003b-4b81-9c0b-a19e20813d23 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:00.361628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-f70dbe29-003b-4b81-9c0b-a19e20813d23 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:00.439197 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:51:00.475632 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70dbe29-003b-4b81-9c0b-a19e20813d23 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:00.486229 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac90668c-712a-4a49-99fe-621a9c52bbd3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:00.487826 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac90668c-712a-4a49-99fe-621a9c52bbd3 service nova] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:00.487826 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac90668c-712a-4a49-99fe-621a9c52bbd3 service nova] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:00.488271 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac90668c-712a-4a49-99fe-621a9c52bbd3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:00.503406 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac90668c-712a-4a49-99fe-621a9c52bbd3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:00.506454 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac90668c-712a-4a49-99fe-621a9c52bbd3 service nova] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:00.523669 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1131/2281] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:51:00 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 166 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:51:00.850549 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd62ff43-9b5b-4fdc-868d-2f39d449e9ad None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:00.891198 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:51:00.937596 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd62ff43-9b5b-4fdc-868d-2f39d449e9ad None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:00.945675 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c486df02-e29a-4d7b-bf55-ca0bd4373f3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:00.946809 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c486df02-e29a-4d7b-bf55-ca0bd4373f3b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:00.947302 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c486df02-e29a-4d7b-bf55-ca0bd4373f3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:00.947805 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c486df02-e29a-4d7b-bf55-ca0bd4373f3b admin admin] PATH_INFO: `/v3/users/838463b8eca244218e50fa343d4b5545` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:00.949989 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c486df02-e29a-4d7b-bf55-ca0bd4373f3b admin admin] RBAC: Authorizing `identity:delete_user(user_id=838463b8eca244218e50fa343d4b5545)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:00.954421 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c486df02-e29a-4d7b-bf55-ca0bd4373f3b admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:01.026216 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c486df02-e29a-4d7b-bf55-ca0bd4373f3b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '838463b8eca244218e50fa343d4b5545'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:01.037418 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c486df02-e29a-4d7b-bf55-ca0bd4373f3b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '838463b8eca244218e50fa343d4b5545'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:01.038674 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c486df02-e29a-4d7b-bf55-ca0bd4373f3b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '838463b8eca244218e50fa343d4b5545'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:01.039466 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c486df02-e29a-4d7b-bf55-ca0bd4373f3b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '838463b8eca244218e50fa343d4b5545'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:01.049598 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c486df02-e29a-4d7b-bf55-ca0bd4373f3b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '838463b8eca244218e50fa343d4b5545'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:01.074835 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1151/2282] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:51:00 2022] DELETE /identity/v3/users/838463b8eca244218e50fa343d4b5545 => generated 0 bytes in 225 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:51:01.080833 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a980338-0ec7-4c9c-a874-f76e3b940a6b None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:01.082066 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a980338-0ec7-4c9c-a874-f76e3b940a6b None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:01.082529 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a980338-0ec7-4c9c-a874-f76e3b940a6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:01.107231 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-0a980338-0ec7-4c9c-a874-f76e3b940a6b None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:51:01.108726 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-0a980338-0ec7-4c9c-a874-f76e3b940a6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:01.181120 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1132/2283] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:51:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:51:01.192741 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c486df02-e29a-4d7b-bf55-ca0bd4373f3b admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:01.194273 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-c486df02-e29a-4d7b-bf55-ca0bd4373f3b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:01.264295 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c486df02-e29a-4d7b-bf55-ca0bd4373f3b admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:01.268254 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8701793-48bf-4f76-8557-2c5d0265e1d8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:01.268883 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8701793-48bf-4f76-8557-2c5d0265e1d8 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:01.269163 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8701793-48bf-4f76-8557-2c5d0265e1d8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:01.269446 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8701793-48bf-4f76-8557-2c5d0265e1d8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:01.280951 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8701793-48bf-4f76-8557-2c5d0265e1d8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:01.282945 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8701793-48bf-4f76-8557-2c5d0265e1d8 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:01.295382 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1152/2284] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:51:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:51:01.618552 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a980338-0ec7-4c9c-a874-f76e3b940a6b None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:01.620761 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-0a980338-0ec7-4c9c-a874-f76e3b940a6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:01.642487 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a980338-0ec7-4c9c-a874-f76e3b940a6b None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:01.650669 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98e33b0f-860c-4fbf-a3db-9f2528e00b00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:01.651674 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e33b0f-860c-4fbf-a3db-9f2528e00b00 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:01.652309 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e33b0f-860c-4fbf-a3db-9f2528e00b00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:01.652616 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98e33b0f-860c-4fbf-a3db-9f2528e00b00 admin admin] PATH_INFO: `/v3/projects/25e4cd851ba947ed96286977236953a6` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:01.654749 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98e33b0f-860c-4fbf-a3db-9f2528e00b00 admin admin] RBAC: Authorizing `identity:delete_project(project_id=25e4cd851ba947ed96286977236953a6)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:01.659577 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98e33b0f-860c-4fbf-a3db-9f2528e00b00 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:01.740211 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-98e33b0f-860c-4fbf-a3db-9f2528e00b00 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '25e4cd851ba947ed96286977236953a6', 'request_id': 'req-98e33b0f-860c-4fbf-a3db-9f2528e00b00'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:01.747719 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-98e33b0f-860c-4fbf-a3db-9f2528e00b00 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '25e4cd851ba947ed96286977236953a6', 'request_id': 'req-98e33b0f-860c-4fbf-a3db-9f2528e00b00'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:01.758085 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-98e33b0f-860c-4fbf-a3db-9f2528e00b00 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '25e4cd851ba947ed96286977236953a6', 'request_id': 'req-98e33b0f-860c-4fbf-a3db-9f2528e00b00'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:01.794370 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-98e33b0f-860c-4fbf-a3db-9f2528e00b00 admin admin] The token cache is being invalidate because project 25e4cd851ba947ed96286977236953a6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:51:01.795920 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-98e33b0f-860c-4fbf-a3db-9f2528e00b00 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:01.797088 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-98e33b0f-860c-4fbf-a3db-9f2528e00b00 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:01.800973 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1133/2285] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:51:01 2022] DELETE /identity/v3/projects/25e4cd851ba947ed96286977236953a6 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:51:01.808027 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1412ef7-7b9f-47cb-9aee-617d3040c798 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:01.808328 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1412ef7-7b9f-47cb-9aee-617d3040c798 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:01.808641 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1412ef7-7b9f-47cb-9aee-617d3040c798 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:01.840201 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-c1412ef7-7b9f-47cb-9aee-617d3040c798 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:51:01.842429 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-c1412ef7-7b9f-47cb-9aee-617d3040c798 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:01.938279 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:51:01.972031 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1153/2286] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:51:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:51:01.980285 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d05754a-702d-40ad-8c1c-43e18c3680ec None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:01.980582 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d05754a-702d-40ad-8c1c-43e18c3680ec None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:01.980863 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d05754a-702d-40ad-8c1c-43e18c3680ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:02.004283 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-8d05754a-702d-40ad-8c1c-43e18c3680ec None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:51:02.005741 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-8d05754a-702d-40ad-8c1c-43e18c3680ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:02.014522 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1134/2287] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:51:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:51:02.024390 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1412ef7-7b9f-47cb-9aee-617d3040c798 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:02.034777 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1412ef7-7b9f-47cb-9aee-617d3040c798 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:02.039976 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd2644ac-e78a-4b26-b291-6796fd3009d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:02.040617 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2644ac-e78a-4b26-b291-6796fd3009d9 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:02.040897 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2644ac-e78a-4b26-b291-6796fd3009d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:02.041171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2644ac-e78a-4b26-b291-6796fd3009d9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:02.041841 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd2644ac-e78a-4b26-b291-6796fd3009d9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:51:02.042151 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd2644ac-e78a-4b26-b291-6796fd3009d9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:02.043855 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd2644ac-e78a-4b26-b291-6796fd3009d9 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:02.057874 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1154/2288] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:51:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:51:02.063902 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d05754a-702d-40ad-8c1c-43e18c3680ec None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:02.082151 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d05754a-702d-40ad-8c1c-43e18c3680ec None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:02.090519 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30ca281-7d8b-4577-9445-f525624df7bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:02.091606 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30ca281-7d8b-4577-9445-f525624df7bb admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:02.092120 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30ca281-7d8b-4577-9445-f525624df7bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:02.092641 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30ca281-7d8b-4577-9445-f525624df7bb admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:02.093823 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f30ca281-7d8b-4577-9445-f525624df7bb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:02.097681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f30ca281-7d8b-4577-9445-f525624df7bb admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:02.150439 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1135/2289] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:51:02 2022] POST /identity/v3/projects => generated 385 bytes in 87 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:51:02.154167 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd2644ac-e78a-4b26-b291-6796fd3009d9 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:02.171380 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd2644ac-e78a-4b26-b291-6796fd3009d9 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:02.248205 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f67a6b7-00f6-4873-bb3a-7c6c12dd58e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:02.248835 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f67a6b7-00f6-4873-bb3a-7c6c12dd58e8 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:02.249113 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f67a6b7-00f6-4873-bb3a-7c6c12dd58e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:02.249394 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f67a6b7-00f6-4873-bb3a-7c6c12dd58e8 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:02.250819 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f67a6b7-00f6-4873-bb3a-7c6c12dd58e8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:02.252693 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f67a6b7-00f6-4873-bb3a-7c6c12dd58e8 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:02.285925 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1155/2290] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:51:02 2022] POST /identity/v3/users => generated 293 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:51:02.290475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30ca281-7d8b-4577-9445-f525624df7bb admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:02.302237 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30ca281-7d8b-4577-9445-f525624df7bb admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:02.307428 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f15bdb9-20db-487e-9ab8-5e9020efa36a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:02.308068 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f15bdb9-20db-487e-9ab8-5e9020efa36a admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:02.308383 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f15bdb9-20db-487e-9ab8-5e9020efa36a admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:02.308657 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f15bdb9-20db-487e-9ab8-5e9020efa36a admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:02.309329 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f15bdb9-20db-487e-9ab8-5e9020efa36a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:02.311030 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f15bdb9-20db-487e-9ab8-5e9020efa36a admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:02.323536 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1136/2291] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:51:02 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:51:02.328340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f67a6b7-00f6-4873-bb3a-7c6c12dd58e8 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:02.340211 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f67a6b7-00f6-4873-bb3a-7c6c12dd58e8 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:02.345942 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa124dff-d699-4617-b933-f494e02e8e83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:02.346637 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa124dff-d699-4617-b933-f494e02e8e83 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:02.346927 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa124dff-d699-4617-b933-f494e02e8e83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:02.347206 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa124dff-d699-4617-b933-f494e02e8e83 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:02.347925 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa124dff-d699-4617-b933-f494e02e8e83 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:02.349654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa124dff-d699-4617-b933-f494e02e8e83 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:02.363677 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1156/2292] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:51:02 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:51:02.371057 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f15bdb9-20db-487e-9ab8-5e9020efa36a admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:02.390890 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f15bdb9-20db-487e-9ab8-5e9020efa36a admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:02.399085 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bd18d5d-8209-4215-ac51-afd2301e0833 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:02.400285 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd18d5d-8209-4215-ac51-afd2301e0833 admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:02.400790 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd18d5d-8209-4215-ac51-afd2301e0833 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:02.401295 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd18d5d-8209-4215-ac51-afd2301e0833 admin admin] PATH_INFO: `/v3/projects/0f9788498e104a7b978a0a7e6abb8ed5/users/2bcbfb425cce4ba8a3f94a3b8cf1ca9b/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:02.432184 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bd18d5d-8209-4215-ac51-afd2301e0833 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0f9788498e104a7b978a0a7e6abb8ed5, user_id=2bcbfb425cce4ba8a3f94a3b8cf1ca9b, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:02.438227 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bd18d5d-8209-4215-ac51-afd2301e0833 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:02.462439 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1137/2293] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:51:02 2022] PUT /identity/v3/projects/0f9788498e104a7b978a0a7e6abb8ed5/users/2bcbfb425cce4ba8a3f94a3b8cf1ca9b/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 93 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:51:02.481199 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312ed1b0-0c46-4261-8175-e690afb8a21b None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:02.481641 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312ed1b0-0c46-4261-8175-e690afb8a21b None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:02.481927 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312ed1b0-0c46-4261-8175-e690afb8a21b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:02.514864 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-312ed1b0-0c46-4261-8175-e690afb8a21b None None] MFA Rules not processed for user `2bcbfb425cce4ba8a3f94a3b8cf1ca9b`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:51:02.516200 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-312ed1b0-0c46-4261-8175-e690afb8a21b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:02.592545 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1157/2294] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:51:02 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:51:02.597718 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bd18d5d-8209-4215-ac51-afd2301e0833 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:02.603799 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:51:02.682981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bd18d5d-8209-4215-ac51-afd2301e0833 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:02.688694 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be77ec52-e7bc-432d-afe5-c7a7fc9ff446 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:02.689331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be77ec52-e7bc-432d-afe5-c7a7fc9ff446 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:02.689642 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be77ec52-e7bc-432d-afe5-c7a7fc9ff446 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:02.689918 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be77ec52-e7bc-432d-afe5-c7a7fc9ff446 admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:02.690621 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be77ec52-e7bc-432d-afe5-c7a7fc9ff446 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:02.692572 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be77ec52-e7bc-432d-afe5-c7a7fc9ff446 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:02.727863 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1138/2295] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:51:02 2022] POST /identity/v3/projects => generated 385 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:51:02.729038 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-312ed1b0-0c46-4261-8175-e690afb8a21b None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:02.741899 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-312ed1b0-0c46-4261-8175-e690afb8a21b None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:02.828065 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:51:02.846415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7f4c71-d824-4dec-b40a-1f59a31b0a3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:02.847348 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7f4c71-d824-4dec-b40a-1f59a31b0a3d admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:02.847761 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7f4c71-d824-4dec-b40a-1f59a31b0a3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:02.848175 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7f4c71-d824-4dec-b40a-1f59a31b0a3d admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:02.849339 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c7f4c71-d824-4dec-b40a-1f59a31b0a3d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:02.853927 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c7f4c71-d824-4dec-b40a-1f59a31b0a3d admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:02.886900 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1158/2296] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:51:02 2022] POST /identity/v3/users => generated 299 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:51:02.891831 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be77ec52-e7bc-432d-afe5-c7a7fc9ff446 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:02.911269 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be77ec52-e7bc-432d-afe5-c7a7fc9ff446 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:02.919217 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3595ed42-ee1d-41e6-b04c-391fb46f79ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:02.920485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3595ed42-ee1d-41e6-b04c-391fb46f79ed admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:02.920953 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3595ed42-ee1d-41e6-b04c-391fb46f79ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:02.921411 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3595ed42-ee1d-41e6-b04c-391fb46f79ed admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:02.923143 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3595ed42-ee1d-41e6-b04c-391fb46f79ed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:02.926189 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3595ed42-ee1d-41e6-b04c-391fb46f79ed admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:02.948587 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1139/2297] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:51:02 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:51:02.952970 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7f4c71-d824-4dec-b40a-1f59a31b0a3d admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:02.964323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7f4c71-d824-4dec-b40a-1f59a31b0a3d admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:02.969933 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83d8f7f6-f9c6-4978-8f91-2a7946cca510 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:02.970670 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d8f7f6-f9c6-4978-8f91-2a7946cca510 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:02.971010 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d8f7f6-f9c6-4978-8f91-2a7946cca510 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:02.971306 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d8f7f6-f9c6-4978-8f91-2a7946cca510 admin admin] PATH_INFO: `/v3/projects/b4db04abd6374e638cf611eb0fb58bfc/users/f16e92af50384ddb85b2df36693518dd/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:02.989395 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83d8f7f6-f9c6-4978-8f91-2a7946cca510 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b4db04abd6374e638cf611eb0fb58bfc, user_id=f16e92af50384ddb85b2df36693518dd, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:02.992206 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83d8f7f6-f9c6-4978-8f91-2a7946cca510 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:03.008887 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1159/2298] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:51:02 2022] PUT /identity/v3/projects/b4db04abd6374e638cf611eb0fb58bfc/users/f16e92af50384ddb85b2df36693518dd/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:51:03.013464 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3595ed42-ee1d-41e6-b04c-391fb46f79ed admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:03.025248 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3595ed42-ee1d-41e6-b04c-391fb46f79ed admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:03.126365 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5918be3f-7861-4e81-879e-a36a7043cfef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:03.127237 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5918be3f-7861-4e81-879e-a36a7043cfef admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:03.127237 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5918be3f-7861-4e81-879e-a36a7043cfef admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:03.127574 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5918be3f-7861-4e81-879e-a36a7043cfef admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:03.128131 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5918be3f-7861-4e81-879e-a36a7043cfef admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:03.129779 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5918be3f-7861-4e81-879e-a36a7043cfef admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:03.141566 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1140/2299] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:51:03 2022] GET /identity/v3/roles => generated 1420 bytes in 129 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:51:03.146177 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83d8f7f6-f9c6-4978-8f91-2a7946cca510 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:03.158641 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83d8f7f6-f9c6-4978-8f91-2a7946cca510 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:03.163639 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1c0281-3403-448a-86a7-f347120f42b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:03.164333 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1c0281-3403-448a-86a7-f347120f42b2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:03.164617 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1c0281-3403-448a-86a7-f347120f42b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:03.164899 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1c0281-3403-448a-86a7-f347120f42b2 admin admin] PATH_INFO: `/v3/projects/b4db04abd6374e638cf611eb0fb58bfc/users/f16e92af50384ddb85b2df36693518dd/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:03.167058 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e1c0281-3403-448a-86a7-f347120f42b2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b4db04abd6374e638cf611eb0fb58bfc, user_id=f16e92af50384ddb85b2df36693518dd, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:03.169781 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e1c0281-3403-448a-86a7-f347120f42b2 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:03.184139 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1160/2300] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:51:03 2022] PUT /identity/v3/projects/b4db04abd6374e638cf611eb0fb58bfc/users/f16e92af50384ddb85b2df36693518dd/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:51:03.201339 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d56c9bc-c820-4368-84b2-1d431a60b483 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:03.201687 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d56c9bc-c820-4368-84b2-1d431a60b483 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:03.201985 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d56c9bc-c820-4368-84b2-1d431a60b483 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:03.245967 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-9d56c9bc-c820-4368-84b2-1d431a60b483 None None] MFA Rules not processed for user `f16e92af50384ddb85b2df36693518dd`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:51:03.247487 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-9d56c9bc-c820-4368-84b2-1d431a60b483 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:03.353110 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:51:03.367366 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1141/2301] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:51:03 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:51:03.380058 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1c0281-3403-448a-86a7-f347120f42b2 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:03.382782 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-2e1c0281-3403-448a-86a7-f347120f42b2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:03.474183 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1c0281-3403-448a-86a7-f347120f42b2 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:03.478087 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14790eb4-e72b-4ad2-b1cc-65a02d667d95 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:03.478768 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14790eb4-e72b-4ad2-b1cc-65a02d667d95 service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:03.479050 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14790eb4-e72b-4ad2-b1cc-65a02d667d95 service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:03.479326 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14790eb4-e72b-4ad2-b1cc-65a02d667d95 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:03.490347 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14790eb4-e72b-4ad2-b1cc-65a02d667d95 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:03.491951 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14790eb4-e72b-4ad2-b1cc-65a02d667d95 service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:03.503621 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1161/2302] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:51:03 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:51:03.538132 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d56c9bc-c820-4368-84b2-1d431a60b483 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:03.619318 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d56c9bc-c820-4368-84b2-1d431a60b483 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:03.624158 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c3f868-1c8c-41e4-be13-73e08c72a0de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:03.625393 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c3f868-1c8c-41e4-be13-73e08c72a0de admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:03.625674 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c3f868-1c8c-41e4-be13-73e08c72a0de admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:03.625956 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c3f868-1c8c-41e4-be13-73e08c72a0de admin admin] PATH_INFO: `/v3/users/2bcbfb425cce4ba8a3f94a3b8cf1ca9b` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:03.627388 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28c3f868-1c8c-41e4-be13-73e08c72a0de admin admin] RBAC: Authorizing `identity:delete_user(user_id=2bcbfb425cce4ba8a3f94a3b8cf1ca9b)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:03.630209 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28c3f868-1c8c-41e4-be13-73e08c72a0de admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:03.698694 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-28c3f868-1c8c-41e4-be13-73e08c72a0de admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2bcbfb425cce4ba8a3f94a3b8cf1ca9b'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:03.708435 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-28c3f868-1c8c-41e4-be13-73e08c72a0de admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2bcbfb425cce4ba8a3f94a3b8cf1ca9b'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:03.709327 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-28c3f868-1c8c-41e4-be13-73e08c72a0de admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2bcbfb425cce4ba8a3f94a3b8cf1ca9b'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:03.710164 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-28c3f868-1c8c-41e4-be13-73e08c72a0de admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2bcbfb425cce4ba8a3f94a3b8cf1ca9b'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:03.721316 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-28c3f868-1c8c-41e4-be13-73e08c72a0de admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2bcbfb425cce4ba8a3f94a3b8cf1ca9b'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:03.747703 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1142/2303] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:51:03 2022] DELETE /identity/v3/users/2bcbfb425cce4ba8a3f94a3b8cf1ca9b => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:51:03.754012 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783f22ac-de3a-48f6-bed4-f380ae24fbeb None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:03.754362 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783f22ac-de3a-48f6-bed4-f380ae24fbeb None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:03.754730 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783f22ac-de3a-48f6-bed4-f380ae24fbeb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:03.776529 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-783f22ac-de3a-48f6-bed4-f380ae24fbeb None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:51:03.777760 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-783f22ac-de3a-48f6-bed4-f380ae24fbeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:03.865800 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1162/2304] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:51:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:51:03.877229 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c3f868-1c8c-41e4-be13-73e08c72a0de admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:03.879557 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-28c3f868-1c8c-41e4-be13-73e08c72a0de admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:03.981558 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c3f868-1c8c-41e4-be13-73e08c72a0de admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:03.986996 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37b6d33-7ce3-4cca-a76e-5a0cfa2726c6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:03.987823 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37b6d33-7ce3-4cca-a76e-5a0cfa2726c6 service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:03.988239 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37b6d33-7ce3-4cca-a76e-5a0cfa2726c6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:03.988799 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b37b6d33-7ce3-4cca-a76e-5a0cfa2726c6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:04.000728 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b37b6d33-7ce3-4cca-a76e-5a0cfa2726c6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:04.002321 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b37b6d33-7ce3-4cca-a76e-5a0cfa2726c6 service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:04.018280 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1143/2305] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:51:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:51:04.642706 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-783f22ac-de3a-48f6-bed4-f380ae24fbeb None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:04.644688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-783f22ac-de3a-48f6-bed4-f380ae24fbeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:04.659105 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-783f22ac-de3a-48f6-bed4-f380ae24fbeb None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:04.664959 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8925084a-d868-4c24-af86-bf4122948d96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:04.665942 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8925084a-d868-4c24-af86-bf4122948d96 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:04.666347 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8925084a-d868-4c24-af86-bf4122948d96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:04.667477 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8925084a-d868-4c24-af86-bf4122948d96 admin admin] PATH_INFO: `/v3/projects/0f9788498e104a7b978a0a7e6abb8ed5` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:04.668730 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8925084a-d868-4c24-af86-bf4122948d96 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0f9788498e104a7b978a0a7e6abb8ed5)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:04.672213 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8925084a-d868-4c24-af86-bf4122948d96 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:04.911325 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=92,keystone:INSERT=5,keystone:DELETE=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:51:04.919479 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-8925084a-d868-4c24-af86-bf4122948d96 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0f9788498e104a7b978a0a7e6abb8ed5', 'request_id': 'req-8925084a-d868-4c24-af86-bf4122948d96'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:04.928538 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-8925084a-d868-4c24-af86-bf4122948d96 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0f9788498e104a7b978a0a7e6abb8ed5', 'request_id': 'req-8925084a-d868-4c24-af86-bf4122948d96'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:04.939026 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-8925084a-d868-4c24-af86-bf4122948d96 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0f9788498e104a7b978a0a7e6abb8ed5', 'request_id': 'req-8925084a-d868-4c24-af86-bf4122948d96'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:04.955099 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-8925084a-d868-4c24-af86-bf4122948d96 admin admin] The token cache is being invalidate because project 0f9788498e104a7b978a0a7e6abb8ed5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:51:04.955711 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-8925084a-d868-4c24-af86-bf4122948d96 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:04.956736 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-8925084a-d868-4c24-af86-bf4122948d96 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:04.958687 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1163/2306] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:51:04 2022] DELETE /identity/v3/projects/0f9788498e104a7b978a0a7e6abb8ed5 => generated 0 bytes in 319 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:51:04.967193 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37b6d33-7ce3-4cca-a76e-5a0cfa2726c6 service neutron] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:04.981174 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b37b6d33-7ce3-4cca-a76e-5a0cfa2726c6 service neutron] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:05.020668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:51:05.089020 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02b36199-31d9-414c-9372-026aab62a5e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:05.089995 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b36199-31d9-414c-9372-026aab62a5e5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:05.090478 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b36199-31d9-414c-9372-026aab62a5e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:05.091316 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b36199-31d9-414c-9372-026aab62a5e5 admin admin] PATH_INFO: `/v3/users/f16e92af50384ddb85b2df36693518dd` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:05.094244 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02b36199-31d9-414c-9372-026aab62a5e5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f16e92af50384ddb85b2df36693518dd)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:05.097862 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02b36199-31d9-414c-9372-026aab62a5e5 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:05.196116 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-02b36199-31d9-414c-9372-026aab62a5e5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f16e92af50384ddb85b2df36693518dd'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:05.208422 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-02b36199-31d9-414c-9372-026aab62a5e5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f16e92af50384ddb85b2df36693518dd'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:05.210082 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-02b36199-31d9-414c-9372-026aab62a5e5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f16e92af50384ddb85b2df36693518dd'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:05.211417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-02b36199-31d9-414c-9372-026aab62a5e5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f16e92af50384ddb85b2df36693518dd'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:05.227078 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-02b36199-31d9-414c-9372-026aab62a5e5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f16e92af50384ddb85b2df36693518dd'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:05.261288 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1144/2307] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:51:04 2022] DELETE /identity/v3/users/f16e92af50384ddb85b2df36693518dd => generated 0 bytes in 293 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:51:05.718086 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8925084a-d868-4c24-af86-bf4122948d96 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:05.720628 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-8925084a-d868-4c24-af86-bf4122948d96 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:05.742878 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8925084a-d868-4c24-af86-bf4122948d96 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:05.843764 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96486161-89bb-4459-aa9a-2eb82b4135b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:05.844837 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96486161-89bb-4459-aa9a-2eb82b4135b9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:05.845294 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96486161-89bb-4459-aa9a-2eb82b4135b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:05.845750 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96486161-89bb-4459-aa9a-2eb82b4135b9 admin admin] PATH_INFO: `/v3/projects/b4db04abd6374e638cf611eb0fb58bfc` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:05.847998 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96486161-89bb-4459-aa9a-2eb82b4135b9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b4db04abd6374e638cf611eb0fb58bfc)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:05.851670 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96486161-89bb-4459-aa9a-2eb82b4135b9 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:05.931607 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-96486161-89bb-4459-aa9a-2eb82b4135b9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b4db04abd6374e638cf611eb0fb58bfc', 'request_id': 'req-96486161-89bb-4459-aa9a-2eb82b4135b9'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:05.947388 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-96486161-89bb-4459-aa9a-2eb82b4135b9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b4db04abd6374e638cf611eb0fb58bfc', 'request_id': 'req-96486161-89bb-4459-aa9a-2eb82b4135b9'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:05.965700 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-96486161-89bb-4459-aa9a-2eb82b4135b9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b4db04abd6374e638cf611eb0fb58bfc', 'request_id': 'req-96486161-89bb-4459-aa9a-2eb82b4135b9'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:06.010136 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-96486161-89bb-4459-aa9a-2eb82b4135b9 admin admin] The token cache is being invalidate because project b4db04abd6374e638cf611eb0fb58bfc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:51:06.010870 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-96486161-89bb-4459-aa9a-2eb82b4135b9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:06.012220 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-96486161-89bb-4459-aa9a-2eb82b4135b9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:06.015424 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1164/2308] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:51:05 2022] DELETE /identity/v3/projects/b4db04abd6374e638cf611eb0fb58bfc => generated 0 bytes in 298 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:51:06.023415 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd281b1-b341-47b7-89c5-548692013021 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:06.024340 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd281b1-b341-47b7-89c5-548692013021 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:06.024340 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd281b1-b341-47b7-89c5-548692013021 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:06.060359 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-4fd281b1-b341-47b7-89c5-548692013021 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:51:06.063786 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-4fd281b1-b341-47b7-89c5-548692013021 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:06.167166 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1145/2309] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:51:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:51:06.177892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d8d7db-9d0c-4630-a5d1-ba61cd05e09e None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:06.178198 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d8d7db-9d0c-4630-a5d1-ba61cd05e09e None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:06.178464 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d8d7db-9d0c-4630-a5d1-ba61cd05e09e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:06.215632 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-e8d8d7db-9d0c-4630-a5d1-ba61cd05e09e None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:51:06.216915 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e8d8d7db-9d0c-4630-a5d1-ba61cd05e09e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:06.225280 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1165/2310] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:51:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:51:06.230683 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd281b1-b341-47b7-89c5-548692013021 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:06.241502 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd281b1-b341-47b7-89c5-548692013021 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:06.246423 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e99eb56-2ca9-4eb7-8b0b-6b7ae88a079e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:06.247061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e99eb56-2ca9-4eb7-8b0b-6b7ae88a079e admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:06.247342 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e99eb56-2ca9-4eb7-8b0b-6b7ae88a079e admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:06.247614 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e99eb56-2ca9-4eb7-8b0b-6b7ae88a079e admin admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:06.248312 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e99eb56-2ca9-4eb7-8b0b-6b7ae88a079e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:51:06.248618 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e99eb56-2ca9-4eb7-8b0b-6b7ae88a079e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:06.250976 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e99eb56-2ca9-4eb7-8b0b-6b7ae88a079e admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:06.269038 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1146/2311] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:51:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:51:06.276981 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d8d7db-9d0c-4630-a5d1-ba61cd05e09e None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:06.291720 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d8d7db-9d0c-4630-a5d1-ba61cd05e09e None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:06.298196 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c436fa3-38da-4712-a2eb-5cc8b30cc1d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:06.299099 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c436fa3-38da-4712-a2eb-5cc8b30cc1d1 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:06.299441 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c436fa3-38da-4712-a2eb-5cc8b30cc1d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:06.299819 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c436fa3-38da-4712-a2eb-5cc8b30cc1d1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:06.300563 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c436fa3-38da-4712-a2eb-5cc8b30cc1d1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:06.303022 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c436fa3-38da-4712-a2eb-5cc8b30cc1d1 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:06.339575 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1166/2312] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:51:06 2022] POST /identity/v3/projects => generated 383 bytes in 60 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:51:06.341643 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e99eb56-2ca9-4eb7-8b0b-6b7ae88a079e admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:06.355950 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e99eb56-2ca9-4eb7-8b0b-6b7ae88a079e admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:06.407715 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:51:06.467733 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c92b102-2e34-4320-9621-07495bb54703 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:06.469179 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c92b102-2e34-4320-9621-07495bb54703 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:06.469545 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c92b102-2e34-4320-9621-07495bb54703 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:06.469936 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c92b102-2e34-4320-9621-07495bb54703 admin admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:06.470798 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c92b102-2e34-4320-9621-07495bb54703 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:06.473272 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c92b102-2e34-4320-9621-07495bb54703 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:06.500577 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1147/2313] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:51:06 2022] POST /identity/v3/users => generated 292 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:51:06.505679 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c436fa3-38da-4712-a2eb-5cc8b30cc1d1 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:06.517917 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c436fa3-38da-4712-a2eb-5cc8b30cc1d1 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:06.525436 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8eb5a9-a7c5-4f86-baf0-33ebc703686a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:06.526702 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8eb5a9-a7c5-4f86-baf0-33ebc703686a admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:06.527309 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8eb5a9-a7c5-4f86-baf0-33ebc703686a admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:06.527798 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8eb5a9-a7c5-4f86-baf0-33ebc703686a admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:06.529067 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8eb5a9-a7c5-4f86-baf0-33ebc703686a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:06.532752 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8eb5a9-a7c5-4f86-baf0-33ebc703686a admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:06.560415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1167/2314] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:51:06 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:51:06.561953 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c92b102-2e34-4320-9621-07495bb54703 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:06.577956 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c92b102-2e34-4320-9621-07495bb54703 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:06.584630 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7853e92-24c4-4ad8-a506-4e0f616ff527 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:06.585841 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7853e92-24c4-4ad8-a506-4e0f616ff527 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:06.586588 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7853e92-24c4-4ad8-a506-4e0f616ff527 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:06.587852 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7853e92-24c4-4ad8-a506-4e0f616ff527 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:06.588279 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7853e92-24c4-4ad8-a506-4e0f616ff527 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:06.591047 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7853e92-24c4-4ad8-a506-4e0f616ff527 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:06.605780 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1148/2315] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:51:06 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:51:06.612635 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8eb5a9-a7c5-4f86-baf0-33ebc703686a admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:06.632507 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8eb5a9-a7c5-4f86-baf0-33ebc703686a admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:06.641533 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b24f11-77d9-416f-bf36-36f0240001bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:06.642669 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b24f11-77d9-416f-bf36-36f0240001bc admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:06.643190 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b24f11-77d9-416f-bf36-36f0240001bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:06.643693 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b24f11-77d9-416f-bf36-36f0240001bc admin admin] PATH_INFO: `/v3/projects/1b4b3e472c5040739fe8175ca42cc23a/users/5e690571bace4c1c8f1c399f5de5ecfb/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:06.671354 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b24f11-77d9-416f-bf36-36f0240001bc admin admin] RBAC: Authorizing `identity:create_grant(project_id=1b4b3e472c5040739fe8175ca42cc23a, user_id=5e690571bace4c1c8f1c399f5de5ecfb, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:06.675402 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b24f11-77d9-416f-bf36-36f0240001bc admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:06.679238 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=85,keystone:DELETE=11,keystone:INSERT=4 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:51:06.703730 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1168/2316] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:51:06 2022] PUT /identity/v3/projects/1b4b3e472c5040739fe8175ca42cc23a/users/5e690571bace4c1c8f1c399f5de5ecfb/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 83 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:51:06.706955 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afe14bb-e420-473e-82b4-10c087af76f7 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:06.707613 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afe14bb-e420-473e-82b4-10c087af76f7 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:06.708158 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1afe14bb-e420-473e-82b4-10c087af76f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:06.756074 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-1afe14bb-e420-473e-82b4-10c087af76f7 None None] MFA Rules not processed for user `5e690571bace4c1c8f1c399f5de5ecfb`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:51:06.757379 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-1afe14bb-e420-473e-82b4-10c087af76f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:06.852148 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1149/2317] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:51:06 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:51:06.863952 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b24f11-77d9-416f-bf36-36f0240001bc admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:06.866018 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-71b24f11-77d9-416f-bf36-36f0240001bc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:06.931774 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b24f11-77d9-416f-bf36-36f0240001bc admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:06.935798 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83befa1b-fa7a-454e-84cc-9ac24925522c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:06.936557 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83befa1b-fa7a-454e-84cc-9ac24925522c service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:06.936927 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83befa1b-fa7a-454e-84cc-9ac24925522c service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:06.937322 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83befa1b-fa7a-454e-84cc-9ac24925522c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:06.947373 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83befa1b-fa7a-454e-84cc-9ac24925522c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:06.948814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83befa1b-fa7a-454e-84cc-9ac24925522c service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:06.959765 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1169/2318] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:51:06 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:51:06.982373 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afe14bb-e420-473e-82b4-10c087af76f7 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:07.070768 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1afe14bb-e420-473e-82b4-10c087af76f7 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:07.078713 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-411ec62f-cda1-4222-99cc-71081afd62d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:07.079722 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411ec62f-cda1-4222-99cc-71081afd62d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:07.080226 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411ec62f-cda1-4222-99cc-71081afd62d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:07.080728 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411ec62f-cda1-4222-99cc-71081afd62d7 admin admin] PATH_INFO: `/v3/users/5e690571bace4c1c8f1c399f5de5ecfb` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:07.082970 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-411ec62f-cda1-4222-99cc-71081afd62d7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5e690571bace4c1c8f1c399f5de5ecfb)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:07.087336 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-411ec62f-cda1-4222-99cc-71081afd62d7 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:07.153543 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=92,keystone:INSERT=3,keystone:DELETE=5 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:51:07.163984 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-411ec62f-cda1-4222-99cc-71081afd62d7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5e690571bace4c1c8f1c399f5de5ecfb'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:07.174911 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-411ec62f-cda1-4222-99cc-71081afd62d7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5e690571bace4c1c8f1c399f5de5ecfb'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:07.175559 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-411ec62f-cda1-4222-99cc-71081afd62d7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5e690571bace4c1c8f1c399f5de5ecfb'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:07.176242 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-411ec62f-cda1-4222-99cc-71081afd62d7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5e690571bace4c1c8f1c399f5de5ecfb'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:07.186356 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-411ec62f-cda1-4222-99cc-71081afd62d7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5e690571bace4c1c8f1c399f5de5ecfb'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:07.213381 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1150/2319] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:51:06 2022] DELETE /identity/v3/users/5e690571bace4c1c8f1c399f5de5ecfb => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:51:07.219212 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39b5722-5c29-4f98-a32f-15c9d5a9d6de None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:07.219611 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39b5722-5c29-4f98-a32f-15c9d5a9d6de None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:07.219892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c39b5722-5c29-4f98-a32f-15c9d5a9d6de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:07.241328 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-c39b5722-5c29-4f98-a32f-15c9d5a9d6de None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:51:07.242628 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-c39b5722-5c29-4f98-a32f-15c9d5a9d6de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:07.330769 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1170/2320] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:51:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:51:07.342093 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-411ec62f-cda1-4222-99cc-71081afd62d7 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:07.344262 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-411ec62f-cda1-4222-99cc-71081afd62d7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:07.449840 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-411ec62f-cda1-4222-99cc-71081afd62d7 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:07.455854 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d472128f-9151-4983-8645-7d68a83ea70a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:07.456632 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d472128f-9151-4983-8645-7d68a83ea70a service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:07.457007 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d472128f-9151-4983-8645-7d68a83ea70a service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:07.457353 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d472128f-9151-4983-8645-7d68a83ea70a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:07.468386 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d472128f-9151-4983-8645-7d68a83ea70a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:07.469856 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d472128f-9151-4983-8645-7d68a83ea70a service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:07.481890 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1151/2321] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:51:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:51:07.824513 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39b5722-5c29-4f98-a32f-15c9d5a9d6de None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:07.826364 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-c39b5722-5c29-4f98-a32f-15c9d5a9d6de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:07.846721 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c39b5722-5c29-4f98-a32f-15c9d5a9d6de None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:07.852739 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a034ea5a-80e1-4090-9c38-bb1aaa315ba7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:07.853548 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a034ea5a-80e1-4090-9c38-bb1aaa315ba7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:07.853905 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a034ea5a-80e1-4090-9c38-bb1aaa315ba7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:07.854249 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a034ea5a-80e1-4090-9c38-bb1aaa315ba7 admin admin] PATH_INFO: `/v3/projects/1b4b3e472c5040739fe8175ca42cc23a` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:07.856725 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a034ea5a-80e1-4090-9c38-bb1aaa315ba7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1b4b3e472c5040739fe8175ca42cc23a)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:07.861575 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a034ea5a-80e1-4090-9c38-bb1aaa315ba7 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:07.929638 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a034ea5a-80e1-4090-9c38-bb1aaa315ba7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1b4b3e472c5040739fe8175ca42cc23a', 'request_id': 'req-a034ea5a-80e1-4090-9c38-bb1aaa315ba7'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:07.943659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a034ea5a-80e1-4090-9c38-bb1aaa315ba7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1b4b3e472c5040739fe8175ca42cc23a', 'request_id': 'req-a034ea5a-80e1-4090-9c38-bb1aaa315ba7'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:07.961166 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a034ea5a-80e1-4090-9c38-bb1aaa315ba7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1b4b3e472c5040739fe8175ca42cc23a', 'request_id': 'req-a034ea5a-80e1-4090-9c38-bb1aaa315ba7'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:07.977800 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a034ea5a-80e1-4090-9c38-bb1aaa315ba7 admin admin] The token cache is being invalidate because project 1b4b3e472c5040739fe8175ca42cc23a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:51:07.978155 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a034ea5a-80e1-4090-9c38-bb1aaa315ba7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:07.978999 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a034ea5a-80e1-4090-9c38-bb1aaa315ba7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:07.980599 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1171/2322] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:51:07 2022] DELETE /identity/v3/projects/1b4b3e472c5040739fe8175ca42cc23a => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:51:07.986693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e304ba9-e272-4e23-a00f-8203603af694 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:07.987321 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e304ba9-e272-4e23-a00f-8203603af694 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:07.987706 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e304ba9-e272-4e23-a00f-8203603af694 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:08.011446 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-0e304ba9-e272-4e23-a00f-8203603af694 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:51:08.013418 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-0e304ba9-e272-4e23-a00f-8203603af694 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:08.123075 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1152/2323] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:51:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:51:08.125874 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78299dbd-62db-4813-b5f4-0878aa5ec9c3 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:08.125874 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78299dbd-62db-4813-b5f4-0878aa5ec9c3 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:08.126133 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78299dbd-62db-4813-b5f4-0878aa5ec9c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:08.159382 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-78299dbd-62db-4813-b5f4-0878aa5ec9c3 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:51:08.161536 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-78299dbd-62db-4813-b5f4-0878aa5ec9c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:08.176601 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1172/2324] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:51:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:51:08.182037 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e304ba9-e272-4e23-a00f-8203603af694 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:08.193834 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e304ba9-e272-4e23-a00f-8203603af694 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:08.200872 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f74a8c-ff5c-4167-b692-6482b979f4a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:08.201480 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f74a8c-ff5c-4167-b692-6482b979f4a8 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:08.201802 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f74a8c-ff5c-4167-b692-6482b979f4a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:08.202088 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f74a8c-ff5c-4167-b692-6482b979f4a8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:08.202862 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f74a8c-ff5c-4167-b692-6482b979f4a8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:51:08.205480 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f74a8c-ff5c-4167-b692-6482b979f4a8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:08.205480 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13f74a8c-ff5c-4167-b692-6482b979f4a8 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:08.226157 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1153/2325] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:51:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:51:08.230259 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78299dbd-62db-4813-b5f4-0878aa5ec9c3 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:08.240981 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78299dbd-62db-4813-b5f4-0878aa5ec9c3 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:08.251270 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68697ed7-e469-4ffd-923c-563a0d7b3081 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:08.251906 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68697ed7-e469-4ffd-923c-563a0d7b3081 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:08.252185 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68697ed7-e469-4ffd-923c-563a0d7b3081 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:08.252450 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68697ed7-e469-4ffd-923c-563a0d7b3081 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:08.253133 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68697ed7-e469-4ffd-923c-563a0d7b3081 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:08.255165 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68697ed7-e469-4ffd-923c-563a0d7b3081 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:08.279880 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:51:08.299988 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1173/2326] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:51:08 2022] POST /identity/v3/projects => generated 361 bytes in 70 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:51:08.305277 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f74a8c-ff5c-4167-b692-6482b979f4a8 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:08.316570 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13f74a8c-ff5c-4167-b692-6482b979f4a8 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:08.370791 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:DELETE=5,keystone:SELECT=94,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:51:08.400625 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00d3810-0795-4223-8630-6c40954164fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:08.401828 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00d3810-0795-4223-8630-6c40954164fa admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:08.402417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00d3810-0795-4223-8630-6c40954164fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:08.403008 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00d3810-0795-4223-8630-6c40954164fa admin admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:08.404359 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b00d3810-0795-4223-8630-6c40954164fa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:08.408502 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b00d3810-0795-4223-8630-6c40954164fa admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:08.444619 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1154/2327] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:51:08 2022] POST /identity/v3/users => generated 281 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:51:08.447975 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68697ed7-e469-4ffd-923c-563a0d7b3081 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:08.459650 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68697ed7-e469-4ffd-923c-563a0d7b3081 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:08.465432 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83244d84-e22c-421d-adc4-2ce164f4c9d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:08.466218 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83244d84-e22c-421d-adc4-2ce164f4c9d2 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:08.466748 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83244d84-e22c-421d-adc4-2ce164f4c9d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:08.467382 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83244d84-e22c-421d-adc4-2ce164f4c9d2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:08.468255 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83244d84-e22c-421d-adc4-2ce164f4c9d2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:08.470383 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83244d84-e22c-421d-adc4-2ce164f4c9d2 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:08.484598 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1174/2328] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:51:08 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:51:08.488770 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00d3810-0795-4223-8630-6c40954164fa admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:08.502044 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00d3810-0795-4223-8630-6c40954164fa admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:08.508230 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ebff1d-1b00-4f59-8db8-22e5c0e4e026 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:08.509036 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ebff1d-1b00-4f59-8db8-22e5c0e4e026 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:08.509377 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ebff1d-1b00-4f59-8db8-22e5c0e4e026 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:08.509816 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ebff1d-1b00-4f59-8db8-22e5c0e4e026 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:08.510911 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1ebff1d-1b00-4f59-8db8-22e5c0e4e026 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:08.512899 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1ebff1d-1b00-4f59-8db8-22e5c0e4e026 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:08.529844 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1155/2329] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:51:08 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:51:08.531806 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83244d84-e22c-421d-adc4-2ce164f4c9d2 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:08.542780 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83244d84-e22c-421d-adc4-2ce164f4c9d2 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:08.548219 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf0c0a2-50bd-4e08-bf1c-d79dce8efd70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:08.548956 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf0c0a2-50bd-4e08-bf1c-d79dce8efd70 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:08.549249 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf0c0a2-50bd-4e08-bf1c-d79dce8efd70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:08.549523 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf0c0a2-50bd-4e08-bf1c-d79dce8efd70 admin admin] PATH_INFO: `/v3/projects/71eeeb7b2d1e4b9ea4e5a7814bff22de/users/0c44cf9785dd4dcf9523c05aad3fd980/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:08.578457 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf0c0a2-50bd-4e08-bf1c-d79dce8efd70 admin admin] RBAC: Authorizing `identity:create_grant(project_id=71eeeb7b2d1e4b9ea4e5a7814bff22de, user_id=0c44cf9785dd4dcf9523c05aad3fd980, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:08.578457 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf0c0a2-50bd-4e08-bf1c-d79dce8efd70 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:08.604352 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1175/2330] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:51:08 2022] PUT /identity/v3/projects/71eeeb7b2d1e4b9ea4e5a7814bff22de/users/0c44cf9785dd4dcf9523c05aad3fd980/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:51:08.615026 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de54fe18-092d-4582-b67f-a8fc4e414da1 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:08.615310 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de54fe18-092d-4582-b67f-a8fc4e414da1 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:08.615638 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de54fe18-092d-4582-b67f-a8fc4e414da1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:08.649282 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-de54fe18-092d-4582-b67f-a8fc4e414da1 None None] MFA Rules not processed for user `0c44cf9785dd4dcf9523c05aad3fd980`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:51:08.650602 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-de54fe18-092d-4582-b67f-a8fc4e414da1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:08.734602 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1156/2331] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:51:08 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:51:08.772431 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf0c0a2-50bd-4e08-bf1c-d79dce8efd70 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:08.774510 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-cdf0c0a2-50bd-4e08-bf1c-d79dce8efd70 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:08.850989 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf0c0a2-50bd-4e08-bf1c-d79dce8efd70 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:08.857620 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f9f9b34-c6c0-44be-b47e-366a207ecb9c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:08.858507 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9f9b34-c6c0-44be-b47e-366a207ecb9c service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:08.858895 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9f9b34-c6c0-44be-b47e-366a207ecb9c service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:08.859521 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f9f9b34-c6c0-44be-b47e-366a207ecb9c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:08.872066 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f9f9b34-c6c0-44be-b47e-366a207ecb9c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:08.873949 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f9f9b34-c6c0-44be-b47e-366a207ecb9c service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:08.887906 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1176/2332] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:51:08 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:51:08.913132 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de54fe18-092d-4582-b67f-a8fc4e414da1 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:09.016651 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de54fe18-092d-4582-b67f-a8fc4e414da1 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:09.022146 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a839685-0381-4645-8be8-e5cd7d191fcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:09.022852 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a839685-0381-4645-8be8-e5cd7d191fcc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:09.023122 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a839685-0381-4645-8be8-e5cd7d191fcc admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:09.023398 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a839685-0381-4645-8be8-e5cd7d191fcc admin admin] PATH_INFO: `/v3/users/0c44cf9785dd4dcf9523c05aad3fd980` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:09.024675 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a839685-0381-4645-8be8-e5cd7d191fcc admin admin] RBAC: Authorizing `identity:delete_user(user_id=0c44cf9785dd4dcf9523c05aad3fd980)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:09.026733 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a839685-0381-4645-8be8-e5cd7d191fcc admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:09.117436 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-7a839685-0381-4645-8be8-e5cd7d191fcc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0c44cf9785dd4dcf9523c05aad3fd980'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:09.128431 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-7a839685-0381-4645-8be8-e5cd7d191fcc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0c44cf9785dd4dcf9523c05aad3fd980'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:09.130136 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-7a839685-0381-4645-8be8-e5cd7d191fcc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0c44cf9785dd4dcf9523c05aad3fd980'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:09.130900 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-7a839685-0381-4645-8be8-e5cd7d191fcc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0c44cf9785dd4dcf9523c05aad3fd980'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:09.134082 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=88,keystone:INSERT=4,keystone:DELETE=8 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:51:09.144825 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-7a839685-0381-4645-8be8-e5cd7d191fcc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0c44cf9785dd4dcf9523c05aad3fd980'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:09.180132 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1157/2333] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:51:08 2022] DELETE /identity/v3/users/0c44cf9785dd4dcf9523c05aad3fd980 => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:51:09.186077 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071028e7-7d58-4028-9c6b-e4074a24ff91 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:09.186646 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071028e7-7d58-4028-9c6b-e4074a24ff91 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:09.187033 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071028e7-7d58-4028-9c6b-e4074a24ff91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:09.220491 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-071028e7-7d58-4028-9c6b-e4074a24ff91 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:51:09.222579 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-071028e7-7d58-4028-9c6b-e4074a24ff91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:09.341068 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1177/2334] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:51:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:51:09.360871 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a839685-0381-4645-8be8-e5cd7d191fcc admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:09.364677 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-7a839685-0381-4645-8be8-e5cd7d191fcc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:09.472647 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a839685-0381-4645-8be8-e5cd7d191fcc admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:09.478950 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb265884-315f-4d2c-8392-389cf37f7a51 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:09.480125 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb265884-315f-4d2c-8392-389cf37f7a51 service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:09.480991 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb265884-315f-4d2c-8392-389cf37f7a51 service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:09.481175 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb265884-315f-4d2c-8392-389cf37f7a51 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:09.499207 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb265884-315f-4d2c-8392-389cf37f7a51 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:09.502649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb265884-315f-4d2c-8392-389cf37f7a51 service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:09.520650 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1158/2335] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:51:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:51:09.948194 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071028e7-7d58-4028-9c6b-e4074a24ff91 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:09.949929 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-071028e7-7d58-4028-9c6b-e4074a24ff91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:09.963435 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071028e7-7d58-4028-9c6b-e4074a24ff91 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:09.969686 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcc9c6f7-070a-40ab-b37b-16f843ef20cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:09.970536 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc9c6f7-070a-40ab-b37b-16f843ef20cc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:09.971505 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc9c6f7-070a-40ab-b37b-16f843ef20cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:09.971852 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc9c6f7-070a-40ab-b37b-16f843ef20cc admin admin] PATH_INFO: `/v3/projects/71eeeb7b2d1e4b9ea4e5a7814bff22de` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:09.975649 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcc9c6f7-070a-40ab-b37b-16f843ef20cc admin admin] RBAC: Authorizing `identity:delete_project(project_id=71eeeb7b2d1e4b9ea4e5a7814bff22de)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:09.977800 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcc9c6f7-070a-40ab-b37b-16f843ef20cc admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:10.038661 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-dcc9c6f7-070a-40ab-b37b-16f843ef20cc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '71eeeb7b2d1e4b9ea4e5a7814bff22de', 'request_id': 'req-dcc9c6f7-070a-40ab-b37b-16f843ef20cc'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:10.048735 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-dcc9c6f7-070a-40ab-b37b-16f843ef20cc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '71eeeb7b2d1e4b9ea4e5a7814bff22de', 'request_id': 'req-dcc9c6f7-070a-40ab-b37b-16f843ef20cc'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:10.052145 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:INSERT=3,keystone:SELECT=93,keystone:DELETE=4 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:51:10.062339 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-dcc9c6f7-070a-40ab-b37b-16f843ef20cc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '71eeeb7b2d1e4b9ea4e5a7814bff22de', 'request_id': 'req-dcc9c6f7-070a-40ab-b37b-16f843ef20cc'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:10.066232 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504a57bd-0c62-4a7a-a2a0-08587f03b3cb None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:10.066936 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504a57bd-0c62-4a7a-a2a0-08587f03b3cb None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:10.067524 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504a57bd-0c62-4a7a-a2a0-08587f03b3cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:10.079279 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-dcc9c6f7-070a-40ab-b37b-16f843ef20cc admin admin] The token cache is being invalidate because project 71eeeb7b2d1e4b9ea4e5a7814bff22de was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:51:10.079612 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-dcc9c6f7-070a-40ab-b37b-16f843ef20cc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:10.080543 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-dcc9c6f7-070a-40ab-b37b-16f843ef20cc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:10.082564 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1178/2336] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:51:09 2022] DELETE /identity/v3/projects/71eeeb7b2d1e4b9ea4e5a7814bff22de => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:51:10.088118 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2ea0bfb-b712-435f-a59f-6ac468eb20e0 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:10.088629 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2ea0bfb-b712-435f-a59f-6ac468eb20e0 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:10.089221 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2ea0bfb-b712-435f-a59f-6ac468eb20e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:10.093199 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-504a57bd-0c62-4a7a-a2a0-08587f03b3cb None None] MFA Rules not processed for user `e65d298cca0f478eba502edf5bd13954`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:51:10.094511 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-504a57bd-0c62-4a7a-a2a0-08587f03b3cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:10.112598 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-c2ea0bfb-b712-435f-a59f-6ac468eb20e0 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:51:10.113849 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-c2ea0bfb-b712-435f-a59f-6ac468eb20e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:10.164427 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1159/2337] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:51:10 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:51:10.203392 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1179/2338] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:51:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:51:10.211996 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76470ed5-ed64-411d-8d8f-924a9ff91114 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:10.212641 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76470ed5-ed64-411d-8d8f-924a9ff91114 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:10.213233 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76470ed5-ed64-411d-8d8f-924a9ff91114 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:10.236104 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-76470ed5-ed64-411d-8d8f-924a9ff91114 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:51:10.237296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-76470ed5-ed64-411d-8d8f-924a9ff91114 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:10.245574 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1160/2339] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:51:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:51:10.251621 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2ea0bfb-b712-435f-a59f-6ac468eb20e0 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:10.267892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2ea0bfb-b712-435f-a59f-6ac468eb20e0 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:10.273561 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4cbcd9-1f82-4c82-9fc8-e3a9a02f4265 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:10.274203 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4cbcd9-1f82-4c82-9fc8-e3a9a02f4265 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:10.274509 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4cbcd9-1f82-4c82-9fc8-e3a9a02f4265 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:10.274790 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4cbcd9-1f82-4c82-9fc8-e3a9a02f4265 admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:10.275487 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe4cbcd9-1f82-4c82-9fc8-e3a9a02f4265 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:51:10.275791 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe4cbcd9-1f82-4c82-9fc8-e3a9a02f4265 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:10.277432 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe4cbcd9-1f82-4c82-9fc8-e3a9a02f4265 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:10.292265 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1180/2340] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:51:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:51:10.297412 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76470ed5-ed64-411d-8d8f-924a9ff91114 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:10.307896 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76470ed5-ed64-411d-8d8f-924a9ff91114 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:10.313654 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2461853d-ed90-453e-bd2a-009d451866ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:10.314313 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2461853d-ed90-453e-bd2a-009d451866ae admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:10.314651 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2461853d-ed90-453e-bd2a-009d451866ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:10.314930 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2461853d-ed90-453e-bd2a-009d451866ae admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:10.315604 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2461853d-ed90-453e-bd2a-009d451866ae admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:10.317649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2461853d-ed90-453e-bd2a-009d451866ae admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:10.350452 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1161/2341] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:51:10 2022] POST /identity/v3/projects => generated 383 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:51:10.354933 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4cbcd9-1f82-4c82-9fc8-e3a9a02f4265 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:10.368009 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4cbcd9-1f82-4c82-9fc8-e3a9a02f4265 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:10.441754 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8748aaa-8525-42fd-a601-86ff2699b75c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:10.442531 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8748aaa-8525-42fd-a601-86ff2699b75c admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:10.442889 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8748aaa-8525-42fd-a601-86ff2699b75c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:10.443262 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8748aaa-8525-42fd-a601-86ff2699b75c admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:10.444090 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8748aaa-8525-42fd-a601-86ff2699b75c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:10.446264 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8748aaa-8525-42fd-a601-86ff2699b75c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:10.473931 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1181/2342] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:51:10 2022] POST /identity/v3/users => generated 292 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:51:10.475353 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2461853d-ed90-453e-bd2a-009d451866ae admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:10.487649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2461853d-ed90-453e-bd2a-009d451866ae admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:10.492973 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249eba82-7eb0-4647-a93a-f5967fc205d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:10.493616 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249eba82-7eb0-4647-a93a-f5967fc205d3 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:10.493903 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249eba82-7eb0-4647-a93a-f5967fc205d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:10.494174 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-249eba82-7eb0-4647-a93a-f5967fc205d3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:10.494892 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249eba82-7eb0-4647-a93a-f5967fc205d3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:10.496567 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-249eba82-7eb0-4647-a93a-f5967fc205d3 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:10.509083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1162/2343] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:51:10 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:51:10.513587 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8748aaa-8525-42fd-a601-86ff2699b75c admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:10.525848 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8748aaa-8525-42fd-a601-86ff2699b75c admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:10.531524 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-012decf0-1e5d-4255-9fdd-e77dd4ff8e7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:10.532155 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012decf0-1e5d-4255-9fdd-e77dd4ff8e7c admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:10.532443 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012decf0-1e5d-4255-9fdd-e77dd4ff8e7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:10.532715 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-012decf0-1e5d-4255-9fdd-e77dd4ff8e7c admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:10.534449 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-012decf0-1e5d-4255-9fdd-e77dd4ff8e7c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:10.535035 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-012decf0-1e5d-4255-9fdd-e77dd4ff8e7c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:10.547901 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1182/2344] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:51:10 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:51:10.552278 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249eba82-7eb0-4647-a93a-f5967fc205d3 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:10.563113 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-249eba82-7eb0-4647-a93a-f5967fc205d3 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:10.568495 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6de62a-35f0-45c0-80a9-183ea61f434b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:10.569519 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6de62a-35f0-45c0-80a9-183ea61f434b admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:10.569804 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6de62a-35f0-45c0-80a9-183ea61f434b admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:10.570243 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d6de62a-35f0-45c0-80a9-183ea61f434b admin admin] PATH_INFO: `/v3/projects/51ccac114bd54916a1a4a2076ec0ba13/users/0e9fcbbc306f4ee08d52a5ca57427619/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:10.588755 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d6de62a-35f0-45c0-80a9-183ea61f434b admin admin] RBAC: Authorizing `identity:create_grant(project_id=51ccac114bd54916a1a4a2076ec0ba13, user_id=0e9fcbbc306f4ee08d52a5ca57427619, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:10.591531 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d6de62a-35f0-45c0-80a9-183ea61f434b admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:10.609317 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1163/2345] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:51:10 2022] PUT /identity/v3/projects/51ccac114bd54916a1a4a2076ec0ba13/users/0e9fcbbc306f4ee08d52a5ca57427619/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:51:10.629123 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf8e566-39e7-4e89-954d-b6f179a75f9f None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:10.629757 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf8e566-39e7-4e89-954d-b6f179a75f9f None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:10.630110 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf8e566-39e7-4e89-954d-b6f179a75f9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:10.664758 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-2bf8e566-39e7-4e89-954d-b6f179a75f9f None None] MFA Rules not processed for user `0e9fcbbc306f4ee08d52a5ca57427619`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:51:10.666209 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-2bf8e566-39e7-4e89-954d-b6f179a75f9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:10.744016 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:DELETE=3,keystone:SELECT=94,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:51:10.790113 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1183/2346] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:51:10 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:51:10.795868 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6de62a-35f0-45c0-80a9-183ea61f434b admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:10.835021 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=96,keystone:DELETE=2,keystone:INSERT=2 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:51:10.907592 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d6de62a-35f0-45c0-80a9-183ea61f434b admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:10.913059 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-354db599-b020-4ded-b20a-606ac878032e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:10.913779 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354db599-b020-4ded-b20a-606ac878032e admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:10.914116 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354db599-b020-4ded-b20a-606ac878032e admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:10.914522 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354db599-b020-4ded-b20a-606ac878032e admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:10.915263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-354db599-b020-4ded-b20a-606ac878032e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:10.917911 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-354db599-b020-4ded-b20a-606ac878032e admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:10.962486 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1164/2347] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:51:10 2022] POST /identity/v3/projects => generated 385 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:51:10.968208 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf8e566-39e7-4e89-954d-b6f179a75f9f None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:10.982680 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf8e566-39e7-4e89-954d-b6f179a75f9f None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:11.102824 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47494870-d535-44c7-9c08-9ca04a9c6eb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:11.103801 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47494870-d535-44c7-9c08-9ca04a9c6eb5 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:11.104253 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47494870-d535-44c7-9c08-9ca04a9c6eb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:11.104691 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47494870-d535-44c7-9c08-9ca04a9c6eb5 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:11.105770 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47494870-d535-44c7-9c08-9ca04a9c6eb5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:11.109302 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47494870-d535-44c7-9c08-9ca04a9c6eb5 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:11.139122 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1184/2348] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:51:10 2022] POST /identity/v3/users => generated 293 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:51:11.144227 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-354db599-b020-4ded-b20a-606ac878032e admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:11.162781 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-354db599-b020-4ded-b20a-606ac878032e admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:11.171866 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112f667c-54c7-42cc-b3b1-d0a9d52af98c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:11.172953 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112f667c-54c7-42cc-b3b1-d0a9d52af98c admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:11.173650 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112f667c-54c7-42cc-b3b1-d0a9d52af98c admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:11.174328 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112f667c-54c7-42cc-b3b1-d0a9d52af98c admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:11.175526 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112f667c-54c7-42cc-b3b1-d0a9d52af98c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:11.178466 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112f667c-54c7-42cc-b3b1-d0a9d52af98c admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:11.192802 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1165/2349] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:51:11 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:51:11.201419 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47494870-d535-44c7-9c08-9ca04a9c6eb5 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:11.228804 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47494870-d535-44c7-9c08-9ca04a9c6eb5 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:11.238836 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eff6f8f-f0e5-46b4-9208-a83fd9bd54d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:11.239815 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eff6f8f-f0e5-46b4-9208-a83fd9bd54d7 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:11.240275 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eff6f8f-f0e5-46b4-9208-a83fd9bd54d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:11.240711 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eff6f8f-f0e5-46b4-9208-a83fd9bd54d7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:11.242011 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eff6f8f-f0e5-46b4-9208-a83fd9bd54d7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:11.244884 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eff6f8f-f0e5-46b4-9208-a83fd9bd54d7 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:11.275056 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1185/2350] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:51:11 2022] GET /identity/v3/roles => generated 1420 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:51:11.276014 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112f667c-54c7-42cc-b3b1-d0a9d52af98c admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:11.296252 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112f667c-54c7-42cc-b3b1-d0a9d52af98c admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:11.306841 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4dfbacd-c5ad-4a4e-a0e6-0087436ab71a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:11.309107 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4dfbacd-c5ad-4a4e-a0e6-0087436ab71a admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:11.310108 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4dfbacd-c5ad-4a4e-a0e6-0087436ab71a admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:11.310684 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4dfbacd-c5ad-4a4e-a0e6-0087436ab71a admin admin] PATH_INFO: `/v3/projects/9c46aa0827464a6b84262600f64913cc/users/623a0f1fe67b4ea096e1b21e2e5cf3ac/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:11.341304 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4dfbacd-c5ad-4a4e-a0e6-0087436ab71a admin admin] RBAC: Authorizing `identity:create_grant(project_id=9c46aa0827464a6b84262600f64913cc, user_id=623a0f1fe67b4ea096e1b21e2e5cf3ac, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:11.348886 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4dfbacd-c5ad-4a4e-a0e6-0087436ab71a admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:11.382625 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1166/2351] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:51:11 2022] PUT /identity/v3/projects/9c46aa0827464a6b84262600f64913cc/users/623a0f1fe67b4ea096e1b21e2e5cf3ac/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:51:11.392038 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef231609-5cd2-49cf-a398-2b500989710a None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:11.392346 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef231609-5cd2-49cf-a398-2b500989710a None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:11.393004 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef231609-5cd2-49cf-a398-2b500989710a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:11.429785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-ef231609-5cd2-49cf-a398-2b500989710a None None] MFA Rules not processed for user `623a0f1fe67b4ea096e1b21e2e5cf3ac`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:51:11.431430 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ef231609-5cd2-49cf-a398-2b500989710a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:11.546925 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1186/2352] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:51:11 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:51:11.566050 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4dfbacd-c5ad-4a4e-a0e6-0087436ab71a admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:11.665619 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4dfbacd-c5ad-4a4e-a0e6-0087436ab71a admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:11.672140 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf793ed-af92-44dc-b9b8-791fc35ea68f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:11.673008 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf793ed-af92-44dc-b9b8-791fc35ea68f admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:11.673361 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf793ed-af92-44dc-b9b8-791fc35ea68f admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:11.673711 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf793ed-af92-44dc-b9b8-791fc35ea68f admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:11.674479 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccf793ed-af92-44dc-b9b8-791fc35ea68f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:11.676704 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccf793ed-af92-44dc-b9b8-791fc35ea68f admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:11.713388 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1167/2353] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:51:11 2022] POST /identity/v3/projects => generated 381 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:51:11.718049 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef231609-5cd2-49cf-a398-2b500989710a None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:11.730071 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef231609-5cd2-49cf-a398-2b500989710a None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:11.823356 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:51:11.850631 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-476499de-0228-4690-a083-00c9b5fef66b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:11.851373 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476499de-0228-4690-a083-00c9b5fef66b admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:11.851804 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476499de-0228-4690-a083-00c9b5fef66b admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:11.852212 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-476499de-0228-4690-a083-00c9b5fef66b admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:11.853110 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-476499de-0228-4690-a083-00c9b5fef66b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:11.855539 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-476499de-0228-4690-a083-00c9b5fef66b admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:11.886337 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1187/2354] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:51:11 2022] POST /identity/v3/users => generated 297 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:51:11.891723 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf793ed-af92-44dc-b9b8-791fc35ea68f admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:11.904184 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf793ed-af92-44dc-b9b8-791fc35ea68f admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:11.910066 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33d42a5-4bd8-4be4-9c63-0b0345ec4cb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:11.910750 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33d42a5-4bd8-4be4-9c63-0b0345ec4cb8 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:11.911033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33d42a5-4bd8-4be4-9c63-0b0345ec4cb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:11.911311 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33d42a5-4bd8-4be4-9c63-0b0345ec4cb8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:11.911976 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d33d42a5-4bd8-4be4-9c63-0b0345ec4cb8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:11.913602 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d33d42a5-4bd8-4be4-9c63-0b0345ec4cb8 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:11.934268 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1168/2355] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:51:11 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:51:11.939002 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-476499de-0228-4690-a083-00c9b5fef66b admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:11.949823 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-476499de-0228-4690-a083-00c9b5fef66b admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:11.954993 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe477fa-5903-43c4-b6b2-43966fd302f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:11.955732 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe477fa-5903-43c4-b6b2-43966fd302f4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:11.956053 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe477fa-5903-43c4-b6b2-43966fd302f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:11.956467 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe477fa-5903-43c4-b6b2-43966fd302f4 admin admin] PATH_INFO: `/v3/projects/e6657307dd6e4157ab42af9876de4fbc/users/d1603ff01f5c4387b1f6c5f39b508578/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:11.974961 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe477fa-5903-43c4-b6b2-43966fd302f4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e6657307dd6e4157ab42af9876de4fbc, user_id=d1603ff01f5c4387b1f6c5f39b508578, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:11.978829 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe477fa-5903-43c4-b6b2-43966fd302f4 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:11.996629 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1188/2356] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:51:11 2022] PUT /identity/v3/projects/e6657307dd6e4157ab42af9876de4fbc/users/d1603ff01f5c4387b1f6c5f39b508578/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:51:12.001916 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33d42a5-4bd8-4be4-9c63-0b0345ec4cb8 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:12.017394 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33d42a5-4bd8-4be4-9c63-0b0345ec4cb8 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:12.083838 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:51:12.107070 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d33d5e-15bf-4dc3-9564-83e7c82e4695 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:12.107627 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d33d5e-15bf-4dc3-9564-83e7c82e4695 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:12.107900 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d33d5e-15bf-4dc3-9564-83e7c82e4695 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:12.108173 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d33d5e-15bf-4dc3-9564-83e7c82e4695 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:12.108838 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d33d5e-15bf-4dc3-9564-83e7c82e4695 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:12.111316 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d33d5e-15bf-4dc3-9564-83e7c82e4695 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:12.123793 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1169/2357] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:51:11 2022] GET /identity/v3/roles => generated 1420 bytes in 123 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:51:12.128974 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe477fa-5903-43c4-b6b2-43966fd302f4 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:12.140493 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe477fa-5903-43c4-b6b2-43966fd302f4 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:12.145424 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac5f3c0-7070-410c-b891-46995286fb2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:12.146204 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac5f3c0-7070-410c-b891-46995286fb2f admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:12.146384 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac5f3c0-7070-410c-b891-46995286fb2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:12.146753 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac5f3c0-7070-410c-b891-46995286fb2f admin admin] PATH_INFO: `/v3/projects/e6657307dd6e4157ab42af9876de4fbc/users/d1603ff01f5c4387b1f6c5f39b508578/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:12.149000 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aac5f3c0-7070-410c-b891-46995286fb2f admin admin] RBAC: Authorizing `identity:create_grant(project_id=e6657307dd6e4157ab42af9876de4fbc, user_id=d1603ff01f5c4387b1f6c5f39b508578, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:12.151793 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aac5f3c0-7070-410c-b891-46995286fb2f admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:12.163978 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1189/2358] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:51:12 2022] PUT /identity/v3/projects/e6657307dd6e4157ab42af9876de4fbc/users/d1603ff01f5c4387b1f6c5f39b508578/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:51:12.176630 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee59b0c-8917-43c1-9ac5-77bde2a4348b None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:12.177428 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee59b0c-8917-43c1-9ac5-77bde2a4348b None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:12.177428 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee59b0c-8917-43c1-9ac5-77bde2a4348b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:12.225772 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-8ee59b0c-8917-43c1-9ac5-77bde2a4348b None None] MFA Rules not processed for user `d1603ff01f5c4387b1f6c5f39b508578`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:51:12.227937 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-8ee59b0c-8917-43c1-9ac5-77bde2a4348b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:12.329303 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1170/2359] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:51:12 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:51:12.341604 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac5f3c0-7070-410c-b891-46995286fb2f admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:12.344784 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-aac5f3c0-7070-410c-b891-46995286fb2f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:12.450146 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac5f3c0-7070-410c-b891-46995286fb2f admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:12.454747 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1849e6-1e68-40d8-952f-1247b9190ff7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:12.455387 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1849e6-1e68-40d8-952f-1247b9190ff7 service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:12.455688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1849e6-1e68-40d8-952f-1247b9190ff7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:12.455943 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1849e6-1e68-40d8-952f-1247b9190ff7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:12.548890 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e1849e6-1e68-40d8-952f-1247b9190ff7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:12.552426 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e1849e6-1e68-40d8-952f-1247b9190ff7 service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:12.569920 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1190/2360] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:51:12 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 229 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:51:14.521493 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee59b0c-8917-43c1-9ac5-77bde2a4348b None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:14.523842 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-8ee59b0c-8917-43c1-9ac5-77bde2a4348b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:14.549215 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee59b0c-8917-43c1-9ac5-77bde2a4348b None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:14.567763 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6763e972-a3ef-4885-b74f-3a319ee48adc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:14.568656 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6763e972-a3ef-4885-b74f-3a319ee48adc service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:14.569288 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6763e972-a3ef-4885-b74f-3a319ee48adc service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:14.569743 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6763e972-a3ef-4885-b74f-3a319ee48adc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:14.655110 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6763e972-a3ef-4885-b74f-3a319ee48adc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:14.657161 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6763e972-a3ef-4885-b74f-3a319ee48adc service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:14.668038 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1171/2361] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:51:14 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:51:15.498976 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1849e6-1e68-40d8-952f-1247b9190ff7 service cinder] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:15.518276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1849e6-1e68-40d8-952f-1247b9190ff7 service cinder] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:15.522831 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e2191a-4e37-4e36-9af7-6194b608ee98 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:15.524437 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e2191a-4e37-4e36-9af7-6194b608ee98 service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:15.524829 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e2191a-4e37-4e36-9af7-6194b608ee98 service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:15.525537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e2191a-4e37-4e36-9af7-6194b608ee98 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:15.536296 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2e2191a-4e37-4e36-9af7-6194b608ee98 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:15.537678 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2e2191a-4e37-4e36-9af7-6194b608ee98 service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:15.548986 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1191/2362] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:51:15 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:51:19.772286 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6763e972-a3ef-4885-b74f-3a319ee48adc service cinder] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:19.871330 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6763e972-a3ef-4885-b74f-3a319ee48adc service cinder] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:19.876653 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa510e49-d08e-4e6f-9bdf-79d35786166f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:19.877319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa510e49-d08e-4e6f-9bdf-79d35786166f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:19.877600 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa510e49-d08e-4e6f-9bdf-79d35786166f admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:19.877877 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa510e49-d08e-4e6f-9bdf-79d35786166f admin admin] PATH_INFO: `/v3/users/0e9fcbbc306f4ee08d52a5ca57427619` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:19.879226 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa510e49-d08e-4e6f-9bdf-79d35786166f admin admin] RBAC: Authorizing `identity:delete_user(user_id=0e9fcbbc306f4ee08d52a5ca57427619)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:19.881223 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa510e49-d08e-4e6f-9bdf-79d35786166f admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:19.884514 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:51:19.955512 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-fa510e49-d08e-4e6f-9bdf-79d35786166f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0e9fcbbc306f4ee08d52a5ca57427619'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:19.972358 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-fa510e49-d08e-4e6f-9bdf-79d35786166f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0e9fcbbc306f4ee08d52a5ca57427619'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:19.973889 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-fa510e49-d08e-4e6f-9bdf-79d35786166f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0e9fcbbc306f4ee08d52a5ca57427619'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:19.975474 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-fa510e49-d08e-4e6f-9bdf-79d35786166f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0e9fcbbc306f4ee08d52a5ca57427619'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:19.991899 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-fa510e49-d08e-4e6f-9bdf-79d35786166f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0e9fcbbc306f4ee08d52a5ca57427619'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:20.036954 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1172/2363] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:51:19 2022] DELETE /identity/v3/users/0e9fcbbc306f4ee08d52a5ca57427619 => generated 0 bytes in 261 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:51:20.039207 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b2f89f-9223-44e5-8b11-74e4a2e74f9d None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:20.040701 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b2f89f-9223-44e5-8b11-74e4a2e74f9d None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:20.041332 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b2f89f-9223-44e5-8b11-74e4a2e74f9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:20.083158 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-04b2f89f-9223-44e5-8b11-74e4a2e74f9d None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:51:20.085842 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-04b2f89f-9223-44e5-8b11-74e4a2e74f9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:20.168868 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:51:20.204181 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1192/2364] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:51:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:51:20.217573 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa510e49-d08e-4e6f-9bdf-79d35786166f admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:20.219939 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-fa510e49-d08e-4e6f-9bdf-79d35786166f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:20.325428 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa510e49-d08e-4e6f-9bdf-79d35786166f admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:20.332955 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823ab6f1-ddad-4068-851f-77d566184b4b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:20.334712 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823ab6f1-ddad-4068-851f-77d566184b4b service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:20.335329 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823ab6f1-ddad-4068-851f-77d566184b4b service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:20.335895 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823ab6f1-ddad-4068-851f-77d566184b4b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:20.355322 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-823ab6f1-ddad-4068-851f-77d566184b4b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:20.358113 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-823ab6f1-ddad-4068-851f-77d566184b4b service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:20.375743 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1173/2365] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:51:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:51:20.862982 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b2f89f-9223-44e5-8b11-74e4a2e74f9d None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:20.864669 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-04b2f89f-9223-44e5-8b11-74e4a2e74f9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:20.885958 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b2f89f-9223-44e5-8b11-74e4a2e74f9d None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:20.897268 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33db6539-3430-4c16-b07f-907863ea7b9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:20.898386 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33db6539-3430-4c16-b07f-907863ea7b9b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:20.898928 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33db6539-3430-4c16-b07f-907863ea7b9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:20.899630 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33db6539-3430-4c16-b07f-907863ea7b9b admin admin] PATH_INFO: `/v3/projects/51ccac114bd54916a1a4a2076ec0ba13` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:20.901819 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33db6539-3430-4c16-b07f-907863ea7b9b admin admin] RBAC: Authorizing `identity:delete_project(project_id=51ccac114bd54916a1a4a2076ec0ba13)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:20.906226 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33db6539-3430-4c16-b07f-907863ea7b9b admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:20.992817 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-33db6539-3430-4c16-b07f-907863ea7b9b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '51ccac114bd54916a1a4a2076ec0ba13', 'request_id': 'req-33db6539-3430-4c16-b07f-907863ea7b9b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:21.000673 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-33db6539-3430-4c16-b07f-907863ea7b9b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '51ccac114bd54916a1a4a2076ec0ba13', 'request_id': 'req-33db6539-3430-4c16-b07f-907863ea7b9b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:21.010387 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-33db6539-3430-4c16-b07f-907863ea7b9b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '51ccac114bd54916a1a4a2076ec0ba13', 'request_id': 'req-33db6539-3430-4c16-b07f-907863ea7b9b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:21.025790 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-33db6539-3430-4c16-b07f-907863ea7b9b admin admin] The token cache is being invalidate because project 51ccac114bd54916a1a4a2076ec0ba13 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:51:21.026146 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-33db6539-3430-4c16-b07f-907863ea7b9b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:21.026869 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-33db6539-3430-4c16-b07f-907863ea7b9b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:21.028403 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1193/2366] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:51:20 2022] DELETE /identity/v3/projects/51ccac114bd54916a1a4a2076ec0ba13 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:51:21.033853 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823ab6f1-ddad-4068-851f-77d566184b4b service neutron] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:21.046014 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823ab6f1-ddad-4068-851f-77d566184b4b service neutron] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:21.116520 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d9cae1-78c5-4e92-8555-64716056658d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:21.117238 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d9cae1-78c5-4e92-8555-64716056658d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:21.117591 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d9cae1-78c5-4e92-8555-64716056658d admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:21.117954 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d9cae1-78c5-4e92-8555-64716056658d admin admin] PATH_INFO: `/v3/users/623a0f1fe67b4ea096e1b21e2e5cf3ac` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:21.119303 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d9cae1-78c5-4e92-8555-64716056658d admin admin] RBAC: Authorizing `identity:delete_user(user_id=623a0f1fe67b4ea096e1b21e2e5cf3ac)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:21.121619 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d9cae1-78c5-4e92-8555-64716056658d admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:21.177825 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:51:21.214720 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-69d9cae1-78c5-4e92-8555-64716056658d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '623a0f1fe67b4ea096e1b21e2e5cf3ac'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:21.227391 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-69d9cae1-78c5-4e92-8555-64716056658d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '623a0f1fe67b4ea096e1b21e2e5cf3ac'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:21.228569 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-69d9cae1-78c5-4e92-8555-64716056658d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '623a0f1fe67b4ea096e1b21e2e5cf3ac'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:21.229496 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-69d9cae1-78c5-4e92-8555-64716056658d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '623a0f1fe67b4ea096e1b21e2e5cf3ac'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:21.243795 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-69d9cae1-78c5-4e92-8555-64716056658d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '623a0f1fe67b4ea096e1b21e2e5cf3ac'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:21.279324 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1174/2367] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:51:21 2022] DELETE /identity/v3/users/623a0f1fe67b4ea096e1b21e2e5cf3ac => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:51:21.711939 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33db6539-3430-4c16-b07f-907863ea7b9b admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:21.713603 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-33db6539-3430-4c16-b07f-907863ea7b9b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:21.730251 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33db6539-3430-4c16-b07f-907863ea7b9b admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:21.809882 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5708bef-845b-4424-94e1-57f14d08e0f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:21.811029 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5708bef-845b-4424-94e1-57f14d08e0f2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:21.811514 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5708bef-845b-4424-94e1-57f14d08e0f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:21.812490 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5708bef-845b-4424-94e1-57f14d08e0f2 admin admin] PATH_INFO: `/v3/projects/9c46aa0827464a6b84262600f64913cc` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:21.814669 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5708bef-845b-4424-94e1-57f14d08e0f2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9c46aa0827464a6b84262600f64913cc)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:21.818353 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5708bef-845b-4424-94e1-57f14d08e0f2 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:21.908758 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a5708bef-845b-4424-94e1-57f14d08e0f2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9c46aa0827464a6b84262600f64913cc', 'request_id': 'req-a5708bef-845b-4424-94e1-57f14d08e0f2'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:21.921459 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a5708bef-845b-4424-94e1-57f14d08e0f2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9c46aa0827464a6b84262600f64913cc', 'request_id': 'req-a5708bef-845b-4424-94e1-57f14d08e0f2'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:21.933419 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a5708bef-845b-4424-94e1-57f14d08e0f2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9c46aa0827464a6b84262600f64913cc', 'request_id': 'req-a5708bef-845b-4424-94e1-57f14d08e0f2'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:21.939719 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:51:21.948913 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a5708bef-845b-4424-94e1-57f14d08e0f2 admin admin] The token cache is being invalidate because project 9c46aa0827464a6b84262600f64913cc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:51:21.949290 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a5708bef-845b-4424-94e1-57f14d08e0f2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:21.950117 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a5708bef-845b-4424-94e1-57f14d08e0f2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:21.951836 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1194/2368] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:51:21 2022] DELETE /identity/v3/projects/9c46aa0827464a6b84262600f64913cc => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:51:21.957216 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d9cae1-78c5-4e92-8555-64716056658d admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:21.959426 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-69d9cae1-78c5-4e92-8555-64716056658d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:21.983020 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d9cae1-78c5-4e92-8555-64716056658d admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:22.073100 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8906d5e8-623f-41ff-8579-839749555015 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:22.074174 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8906d5e8-623f-41ff-8579-839749555015 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:22.074705 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8906d5e8-623f-41ff-8579-839749555015 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:22.075193 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8906d5e8-623f-41ff-8579-839749555015 admin admin] PATH_INFO: `/v3/users/d1603ff01f5c4387b1f6c5f39b508578` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:22.077279 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8906d5e8-623f-41ff-8579-839749555015 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d1603ff01f5c4387b1f6c5f39b508578)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:22.081206 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8906d5e8-623f-41ff-8579-839749555015 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:22.151671 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-8906d5e8-623f-41ff-8579-839749555015 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd1603ff01f5c4387b1f6c5f39b508578'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:22.161874 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-8906d5e8-623f-41ff-8579-839749555015 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd1603ff01f5c4387b1f6c5f39b508578'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:22.162662 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-8906d5e8-623f-41ff-8579-839749555015 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd1603ff01f5c4387b1f6c5f39b508578'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:22.163619 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-8906d5e8-623f-41ff-8579-839749555015 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd1603ff01f5c4387b1f6c5f39b508578'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:22.178592 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-8906d5e8-623f-41ff-8579-839749555015 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd1603ff01f5c4387b1f6c5f39b508578'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:22.212620 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1175/2369] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:51:21 2022] DELETE /identity/v3/users/d1603ff01f5c4387b1f6c5f39b508578 => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:51:22.606331 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5708bef-845b-4424-94e1-57f14d08e0f2 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:22.608679 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a5708bef-845b-4424-94e1-57f14d08e0f2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:22.624261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5708bef-845b-4424-94e1-57f14d08e0f2 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:22.707171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13e7982-16fc-4751-b0e2-221ac07d06ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:22.708224 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13e7982-16fc-4751-b0e2-221ac07d06ce admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:22.708696 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13e7982-16fc-4751-b0e2-221ac07d06ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:22.709092 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13e7982-16fc-4751-b0e2-221ac07d06ce admin admin] PATH_INFO: `/v3/projects/e6657307dd6e4157ab42af9876de4fbc` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:22.710735 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c13e7982-16fc-4751-b0e2-221ac07d06ce admin admin] RBAC: Authorizing `identity:delete_project(project_id=e6657307dd6e4157ab42af9876de4fbc)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:22.712829 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c13e7982-16fc-4751-b0e2-221ac07d06ce admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:22.800482 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c13e7982-16fc-4751-b0e2-221ac07d06ce admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e6657307dd6e4157ab42af9876de4fbc', 'request_id': 'req-c13e7982-16fc-4751-b0e2-221ac07d06ce'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:22.809491 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c13e7982-16fc-4751-b0e2-221ac07d06ce admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e6657307dd6e4157ab42af9876de4fbc', 'request_id': 'req-c13e7982-16fc-4751-b0e2-221ac07d06ce'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:22.820489 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c13e7982-16fc-4751-b0e2-221ac07d06ce admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e6657307dd6e4157ab42af9876de4fbc', 'request_id': 'req-c13e7982-16fc-4751-b0e2-221ac07d06ce'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:22.837719 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c13e7982-16fc-4751-b0e2-221ac07d06ce admin admin] The token cache is being invalidate because project e6657307dd6e4157ab42af9876de4fbc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:51:22.838161 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c13e7982-16fc-4751-b0e2-221ac07d06ce admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:22.839165 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c13e7982-16fc-4751-b0e2-221ac07d06ce admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:22.841086 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1195/2370] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:51:22 2022] DELETE /identity/v3/projects/e6657307dd6e4157ab42af9876de4fbc => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:51:22.848349 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa27165e-1e22-4da8-819d-7321adb0b24a None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:22.849799 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa27165e-1e22-4da8-819d-7321adb0b24a None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:22.850417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa27165e-1e22-4da8-819d-7321adb0b24a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:22.886606 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-aa27165e-1e22-4da8-819d-7321adb0b24a None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:51:22.888547 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-aa27165e-1e22-4da8-819d-7321adb0b24a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:22.913608 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:DELETE=16,keystone:SELECT=82,keystone:INSERT=2 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:51:22.978201 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1176/2371] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:51:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:51:22.988852 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa7ec25-9858-4658-98b3-7aa8ea29ae02 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:22.989576 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa7ec25-9858-4658-98b3-7aa8ea29ae02 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:22.990152 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa7ec25-9858-4658-98b3-7aa8ea29ae02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:23.024994 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-0fa7ec25-9858-4658-98b3-7aa8ea29ae02 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:51:23.027384 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-0fa7ec25-9858-4658-98b3-7aa8ea29ae02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:23.041995 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1196/2372] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:51:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:51:23.047865 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa27165e-1e22-4da8-819d-7321adb0b24a None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:23.060535 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa27165e-1e22-4da8-819d-7321adb0b24a None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:23.066532 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71ea1ed-444d-4d4c-b50a-dcf816b80208 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:23.067326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71ea1ed-444d-4d4c-b50a-dcf816b80208 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:23.067703 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71ea1ed-444d-4d4c-b50a-dcf816b80208 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:23.068076 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71ea1ed-444d-4d4c-b50a-dcf816b80208 admin admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:23.068958 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b71ea1ed-444d-4d4c-b50a-dcf816b80208 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:51:23.069387 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b71ea1ed-444d-4d4c-b50a-dcf816b80208 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:23.071271 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b71ea1ed-444d-4d4c-b50a-dcf816b80208 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:23.087743 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1177/2373] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:51:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:51:23.094105 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa7ec25-9858-4658-98b3-7aa8ea29ae02 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:23.111397 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa7ec25-9858-4658-98b3-7aa8ea29ae02 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:23.117005 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2767c4e5-21cc-4bdf-b0a4-03967dc7e24f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:23.117941 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2767c4e5-21cc-4bdf-b0a4-03967dc7e24f admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:23.118421 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2767c4e5-21cc-4bdf-b0a4-03967dc7e24f admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:23.118915 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2767c4e5-21cc-4bdf-b0a4-03967dc7e24f admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:23.119984 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2767c4e5-21cc-4bdf-b0a4-03967dc7e24f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:23.122156 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2767c4e5-21cc-4bdf-b0a4-03967dc7e24f admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:23.161950 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1197/2374] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:51:23 2022] POST /identity/v3/projects => generated 369 bytes in 70 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:51:23.166792 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71ea1ed-444d-4d4c-b50a-dcf816b80208 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:23.179465 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71ea1ed-444d-4d4c-b50a-dcf816b80208 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:23.271705 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93502bc1-603a-4561-b5b2-e985dfb179c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:23.272790 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93502bc1-603a-4561-b5b2-e985dfb179c4 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:23.273458 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93502bc1-603a-4561-b5b2-e985dfb179c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:23.273926 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93502bc1-603a-4561-b5b2-e985dfb179c4 admin admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:23.275390 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93502bc1-603a-4561-b5b2-e985dfb179c4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:23.279405 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93502bc1-603a-4561-b5b2-e985dfb179c4 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:23.326847 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1178/2375] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:51:23 2022] POST /identity/v3/users => generated 285 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:51:23.333244 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2767c4e5-21cc-4bdf-b0a4-03967dc7e24f admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:23.352408 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2767c4e5-21cc-4bdf-b0a4-03967dc7e24f admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:23.360773 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0b793a-9f08-4992-be91-842ad1405976 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:23.361779 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0b793a-9f08-4992-be91-842ad1405976 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:23.362267 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0b793a-9f08-4992-be91-842ad1405976 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:23.362773 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0b793a-9f08-4992-be91-842ad1405976 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:23.363905 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b0b793a-9f08-4992-be91-842ad1405976 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:23.366900 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b0b793a-9f08-4992-be91-842ad1405976 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:23.386494 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1198/2376] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:51:23 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:51:23.393270 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93502bc1-603a-4561-b5b2-e985dfb179c4 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:23.411293 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93502bc1-603a-4561-b5b2-e985dfb179c4 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:23.419482 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf8077f-6263-4f05-9b5d-cfec2cb6b5ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:23.420776 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf8077f-6263-4f05-9b5d-cfec2cb6b5ad admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:23.421470 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf8077f-6263-4f05-9b5d-cfec2cb6b5ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:23.421798 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf8077f-6263-4f05-9b5d-cfec2cb6b5ad admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:23.423086 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf8077f-6263-4f05-9b5d-cfec2cb6b5ad admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:23.426002 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf8077f-6263-4f05-9b5d-cfec2cb6b5ad admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:23.446602 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1179/2377] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:51:23 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:51:23.453504 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0b793a-9f08-4992-be91-842ad1405976 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:23.467498 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0b793a-9f08-4992-be91-842ad1405976 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:23.476653 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757ae695-a836-4a7f-94a6-74bd9c071276 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:23.477796 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757ae695-a836-4a7f-94a6-74bd9c071276 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:23.478286 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757ae695-a836-4a7f-94a6-74bd9c071276 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:23.478906 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757ae695-a836-4a7f-94a6-74bd9c071276 admin admin] PATH_INFO: `/v3/projects/d497b05ede294811b843f4a78418606f/users/3474474c291b48018272f163a313424b/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:23.504675 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757ae695-a836-4a7f-94a6-74bd9c071276 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d497b05ede294811b843f4a78418606f, user_id=3474474c291b48018272f163a313424b, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:23.510063 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757ae695-a836-4a7f-94a6-74bd9c071276 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:23.534204 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1199/2378] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:51:23 2022] PUT /identity/v3/projects/d497b05ede294811b843f4a78418606f/users/3474474c291b48018272f163a313424b/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 82 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:51:23.548714 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d37361-81ff-47ea-9dc1-ad565a694417 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:23.549364 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d37361-81ff-47ea-9dc1-ad565a694417 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:23.549982 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d37361-81ff-47ea-9dc1-ad565a694417 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:23.598606 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-c0d37361-81ff-47ea-9dc1-ad565a694417 None None] MFA Rules not processed for user `3474474c291b48018272f163a313424b`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:51:23.600339 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-c0d37361-81ff-47ea-9dc1-ad565a694417 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:23.695824 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1180/2379] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:51:23 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:51:23.710645 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757ae695-a836-4a7f-94a6-74bd9c071276 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:23.713332 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-757ae695-a836-4a7f-94a6-74bd9c071276 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:23.765183 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:51:23.813732 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757ae695-a836-4a7f-94a6-74bd9c071276 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:23.818843 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91fde10e-e2b6-4b5f-96d7-9e0bfea28f60 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4b2ea9aa31445d69e0c1d96b5657708', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:23.819698 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91fde10e-e2b6-4b5f-96d7-9e0bfea28f60 service glance] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:23.819981 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91fde10e-e2b6-4b5f-96d7-9e0bfea28f60 service glance] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:23.820255 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91fde10e-e2b6-4b5f-96d7-9e0bfea28f60 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:23.831040 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91fde10e-e2b6-4b5f-96d7-9e0bfea28f60 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:23.832499 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91fde10e-e2b6-4b5f-96d7-9e0bfea28f60 service glance] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:23.844529 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1200/2380] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:51:23 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:51:29.855683 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1050b6-f423-4153-96e4-949c5904750b None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:29.856339 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1050b6-f423-4153-96e4-949c5904750b None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:29.856339 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1050b6-f423-4153-96e4-949c5904750b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:29.882868 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-ce1050b6-f423-4153-96e4-949c5904750b None None] MFA Rules not processed for user `97c8877c74194a6bbbf25b3c8f5b7dbd`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:51:29.884285 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-ce1050b6-f423-4153-96e4-949c5904750b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:29.903070 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:51:29.982469 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1181/2381] 10.222.0.25 () {58 vars in 1009 bytes} [Fri Oct 28 09:51:29 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:51:30.051175 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91fde10e-e2b6-4b5f-96d7-9e0bfea28f60 service glance] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:30.053408 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-91fde10e-e2b6-4b5f-96d7-9e0bfea28f60 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:30.156832 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91fde10e-e2b6-4b5f-96d7-9e0bfea28f60 service glance] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:30.162283 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c3bd2b-7088-493c-acd5-fbaae2ef4431 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c6cffe0d56e45c4b8fb54fbe5df6094', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:30.162946 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c3bd2b-7088-493c-acd5-fbaae2ef4431 service swift] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:30.163257 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c3bd2b-7088-493c-acd5-fbaae2ef4431 service swift] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:30.163532 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c3bd2b-7088-493c-acd5-fbaae2ef4431 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:30.175148 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c3bd2b-7088-493c-acd5-fbaae2ef4431 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:30.177020 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c3bd2b-7088-493c-acd5-fbaae2ef4431 service swift] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:30.191050 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1201/2382] 10.222.0.25 () {60 vars in 1384 bytes} [Fri Oct 28 09:51:30 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 28 09:51:39.970249 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=17 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:51:40.188975 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=48 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:51:52.412627 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce1050b6-f423-4153-96e4-949c5904750b None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:52.503352 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce1050b6-f423-4153-96e4-949c5904750b None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:52.508766 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ff8b40-a327-489f-a674-562f6b9ff8d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:52.509534 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ff8b40-a327-489f-a674-562f6b9ff8d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:52.509813 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ff8b40-a327-489f-a674-562f6b9ff8d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:52.510089 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ff8b40-a327-489f-a674-562f6b9ff8d8 admin admin] PATH_INFO: `/v3/users/3474474c291b48018272f163a313424b` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:52.511395 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32ff8b40-a327-489f-a674-562f6b9ff8d8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3474474c291b48018272f163a313424b)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:52.513359 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32ff8b40-a327-489f-a674-562f6b9ff8d8 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:52.575470 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-32ff8b40-a327-489f-a674-562f6b9ff8d8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3474474c291b48018272f163a313424b'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:52.585282 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-32ff8b40-a327-489f-a674-562f6b9ff8d8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3474474c291b48018272f163a313424b'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:52.585954 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-32ff8b40-a327-489f-a674-562f6b9ff8d8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3474474c291b48018272f163a313424b'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:52.586590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-32ff8b40-a327-489f-a674-562f6b9ff8d8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3474474c291b48018272f163a313424b'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:52.599253 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-32ff8b40-a327-489f-a674-562f6b9ff8d8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3474474c291b48018272f163a313424b'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:52.627596 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1182/2383] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:51:52 2022] DELETE /identity/v3/users/3474474c291b48018272f163a313424b => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:51:52.632555 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c4d5ce-cb33-4aa0-a005-55cf4defe04d None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:52.633171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c4d5ce-cb33-4aa0-a005-55cf4defe04d None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:52.633171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c4d5ce-cb33-4aa0-a005-55cf4defe04d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:52.653887 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-91c4d5ce-cb33-4aa0-a005-55cf4defe04d None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:51:52.655249 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-91c4d5ce-cb33-4aa0-a005-55cf4defe04d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:52.727771 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1202/2384] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:51:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:51:52.739334 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ff8b40-a327-489f-a674-562f6b9ff8d8 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:52.740794 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-32ff8b40-a327-489f-a674-562f6b9ff8d8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:52.810742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ff8b40-a327-489f-a674-562f6b9ff8d8 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:52.815457 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c6119a-2e50-4fbc-bfb3-b2f747654405 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:52.816094 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c6119a-2e50-4fbc-bfb3-b2f747654405 service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:52.816378 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c6119a-2e50-4fbc-bfb3-b2f747654405 service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:52.816651 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c6119a-2e50-4fbc-bfb3-b2f747654405 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:52.827389 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c6119a-2e50-4fbc-bfb3-b2f747654405 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:52.828860 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c6119a-2e50-4fbc-bfb3-b2f747654405 service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:52.840153 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1183/2385] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:51:52 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:51:53.300432 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c4d5ce-cb33-4aa0-a005-55cf4defe04d None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:53.302819 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-91c4d5ce-cb33-4aa0-a005-55cf4defe04d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:53.316181 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c4d5ce-cb33-4aa0-a005-55cf4defe04d None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:53.322837 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99881d86-c2b4-4988-b022-e8d901ff6b79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:53.323142 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99881d86-c2b4-4988-b022-e8d901ff6b79 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:53.323945 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99881d86-c2b4-4988-b022-e8d901ff6b79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:53.323945 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99881d86-c2b4-4988-b022-e8d901ff6b79 admin admin] PATH_INFO: `/v3/projects/d497b05ede294811b843f4a78418606f` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:53.325108 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99881d86-c2b4-4988-b022-e8d901ff6b79 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d497b05ede294811b843f4a78418606f)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:53.327573 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99881d86-c2b4-4988-b022-e8d901ff6b79 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:53.401715 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-99881d86-c2b4-4988-b022-e8d901ff6b79 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd497b05ede294811b843f4a78418606f', 'request_id': 'req-99881d86-c2b4-4988-b022-e8d901ff6b79'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:53.411019 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-99881d86-c2b4-4988-b022-e8d901ff6b79 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd497b05ede294811b843f4a78418606f', 'request_id': 'req-99881d86-c2b4-4988-b022-e8d901ff6b79'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:53.421447 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-99881d86-c2b4-4988-b022-e8d901ff6b79 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd497b05ede294811b843f4a78418606f', 'request_id': 'req-99881d86-c2b4-4988-b022-e8d901ff6b79'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:53.434456 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-99881d86-c2b4-4988-b022-e8d901ff6b79 admin admin] The token cache is being invalidate because project d497b05ede294811b843f4a78418606f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:51:53.434760 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-99881d86-c2b4-4988-b022-e8d901ff6b79 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:53.435432 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-99881d86-c2b4-4988-b022-e8d901ff6b79 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:51:53.436981 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1203/2386] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:51:53 2022] DELETE /identity/v3/projects/d497b05ede294811b843f4a78418606f => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:51:53.442385 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0fe63d8-68e7-4791-8f89-5788ca3dc9d6 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:53.443038 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0fe63d8-68e7-4791-8f89-5788ca3dc9d6 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:53.443038 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0fe63d8-68e7-4791-8f89-5788ca3dc9d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:53.468573 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-c0fe63d8-68e7-4791-8f89-5788ca3dc9d6 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:51:53.470236 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-c0fe63d8-68e7-4791-8f89-5788ca3dc9d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:53.492822 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:51:53.551224 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1184/2387] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:51:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:51:53.559535 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a0a384-d66d-404c-9bdc-ab8d85e998dd None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:53.559980 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a0a384-d66d-404c-9bdc-ab8d85e998dd None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:53.560142 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a0a384-d66d-404c-9bdc-ab8d85e998dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:53.583713 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-b6a0a384-d66d-404c-9bdc-ab8d85e998dd None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:51:53.584970 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-b6a0a384-d66d-404c-9bdc-ab8d85e998dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:53.593401 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1204/2388] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:51:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:51:53.598053 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0fe63d8-68e7-4791-8f89-5788ca3dc9d6 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:53.609189 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0fe63d8-68e7-4791-8f89-5788ca3dc9d6 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:53.613963 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4505db1-bac8-4c65-b8af-50a1cfae56c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:53.614628 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4505db1-bac8-4c65-b8af-50a1cfae56c2 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:53.614914 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4505db1-bac8-4c65-b8af-50a1cfae56c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:53.615195 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4505db1-bac8-4c65-b8af-50a1cfae56c2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:53.615863 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4505db1-bac8-4c65-b8af-50a1cfae56c2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:51:53.616174 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4505db1-bac8-4c65-b8af-50a1cfae56c2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:53.617852 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4505db1-bac8-4c65-b8af-50a1cfae56c2 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:53.632264 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1185/2389] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:51:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:51:53.636998 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a0a384-d66d-404c-9bdc-ab8d85e998dd None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:53.648947 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a0a384-d66d-404c-9bdc-ab8d85e998dd None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:53.654041 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af57a802-1ec6-4b26-bc01-621674cb6984 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:53.654964 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af57a802-1ec6-4b26-bc01-621674cb6984 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:53.655366 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af57a802-1ec6-4b26-bc01-621674cb6984 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:53.655651 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af57a802-1ec6-4b26-bc01-621674cb6984 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:53.656331 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af57a802-1ec6-4b26-bc01-621674cb6984 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:53.658320 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af57a802-1ec6-4b26-bc01-621674cb6984 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:53.696298 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1205/2390] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:51:53 2022] POST /identity/v3/projects => generated 371 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:51:53.701118 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4505db1-bac8-4c65-b8af-50a1cfae56c2 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:53.716341 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4505db1-bac8-4c65-b8af-50a1cfae56c2 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:53.783667 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af268949-ea90-4ea2-a097-446d10ee2ed9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:53.784195 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af268949-ea90-4ea2-a097-446d10ee2ed9 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:53.784477 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af268949-ea90-4ea2-a097-446d10ee2ed9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:53.784755 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af268949-ea90-4ea2-a097-446d10ee2ed9 admin admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:53.785402 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af268949-ea90-4ea2-a097-446d10ee2ed9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:53.787405 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af268949-ea90-4ea2-a097-446d10ee2ed9 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:53.813707 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1186/2391] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:51:53 2022] POST /identity/v3/users => generated 286 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:51:53.818649 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af57a802-1ec6-4b26-bc01-621674cb6984 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:53.830231 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af57a802-1ec6-4b26-bc01-621674cb6984 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:53.835251 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ec434a-a83b-4fd3-983f-758c3d8b938d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:53.835875 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ec434a-a83b-4fd3-983f-758c3d8b938d admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:53.836153 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ec434a-a83b-4fd3-983f-758c3d8b938d admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:53.836424 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ec434a-a83b-4fd3-983f-758c3d8b938d admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:53.837084 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36ec434a-a83b-4fd3-983f-758c3d8b938d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:53.838742 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36ec434a-a83b-4fd3-983f-758c3d8b938d admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:53.850569 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1206/2392] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:51:53 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:51:53.855390 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af268949-ea90-4ea2-a097-446d10ee2ed9 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:53.868729 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af268949-ea90-4ea2-a097-446d10ee2ed9 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:53.874606 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adc3182b-38f1-4ea6-81ef-6b8da2cdbb1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:53.875239 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc3182b-38f1-4ea6-81ef-6b8da2cdbb1a admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:53.875524 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc3182b-38f1-4ea6-81ef-6b8da2cdbb1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:53.875797 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adc3182b-38f1-4ea6-81ef-6b8da2cdbb1a admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:53.876477 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adc3182b-38f1-4ea6-81ef-6b8da2cdbb1a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:53.878123 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adc3182b-38f1-4ea6-81ef-6b8da2cdbb1a admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:53.890422 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1187/2393] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:51:53 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:51:53.896168 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ec434a-a83b-4fd3-983f-758c3d8b938d admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:53.907380 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ec434a-a83b-4fd3-983f-758c3d8b938d admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:53.912332 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79247406-bb7e-4219-92c4-702569d7081f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:53.913003 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79247406-bb7e-4219-92c4-702569d7081f admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:53.913281 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79247406-bb7e-4219-92c4-702569d7081f admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:53.913555 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79247406-bb7e-4219-92c4-702569d7081f admin admin] PATH_INFO: `/v3/projects/937954dbeac043c29259ada2924c4953/users/f14c3f34010b41ef9be2b9fd2aa98cd9/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:53.931096 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79247406-bb7e-4219-92c4-702569d7081f admin admin] RBAC: Authorizing `identity:create_grant(project_id=937954dbeac043c29259ada2924c4953, user_id=f14c3f34010b41ef9be2b9fd2aa98cd9, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:53.933839 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79247406-bb7e-4219-92c4-702569d7081f admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:53.949561 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1207/2394] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:51:53 2022] PUT /identity/v3/projects/937954dbeac043c29259ada2924c4953/users/f14c3f34010b41ef9be2b9fd2aa98cd9/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:51:53.955562 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9970fca6-7e61-44eb-a088-d4035885a91e None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:53.955995 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9970fca6-7e61-44eb-a088-d4035885a91e None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:53.956366 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9970fca6-7e61-44eb-a088-d4035885a91e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:53.979931 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-9970fca6-7e61-44eb-a088-d4035885a91e None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:51:53.981997 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-9970fca6-7e61-44eb-a088-d4035885a91e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:54.065626 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1188/2395] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:51:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:51:54.076132 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79247406-bb7e-4219-92c4-702569d7081f admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:51:54.078783 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-79247406-bb7e-4219-92c4-702569d7081f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:54.104884 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:51:54.151046 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79247406-bb7e-4219-92c4-702569d7081f admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:51:54.155023 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f205e3d-0a5c-44ba-9586-97ef99132fb7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:51:54.155662 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f205e3d-0a5c-44ba-9586-97ef99132fb7 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:54.155947 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f205e3d-0a5c-44ba-9586-97ef99132fb7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:54.156220 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f205e3d-0a5c-44ba-9586-97ef99132fb7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:54.167365 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f205e3d-0a5c-44ba-9586-97ef99132fb7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:51:54.168839 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f205e3d-0a5c-44ba-9586-97ef99132fb7 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:51:54.180739 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1208/2396] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:51:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:51:59.808269 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d278c838-b6a0-4467-9a33-339e25867acb None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:51:59.809134 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d278c838-b6a0-4467-9a33-339e25867acb None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:51:59.809134 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d278c838-b6a0-4467-9a33-339e25867acb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:51:59.862473 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-d278c838-b6a0-4467-9a33-339e25867acb None None] MFA Rules not processed for user `f14c3f34010b41ef9be2b9fd2aa98cd9`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:51:59.864775 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-d278c838-b6a0-4467-9a33-339e25867acb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:51:59.900812 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:52:00.010237 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1189/2397] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:51:59 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:52:00.017120 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f205e3d-0a5c-44ba-9586-97ef99132fb7 service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:00.051101 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f205e3d-0a5c-44ba-9586-97ef99132fb7 service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:00.060855 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad64afa0-477b-4d09-bb44-e50ed93e07a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:00.062356 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad64afa0-477b-4d09-bb44-e50ed93e07a7 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:00.062997 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad64afa0-477b-4d09-bb44-e50ed93e07a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:00.063429 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad64afa0-477b-4d09-bb44-e50ed93e07a7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:00.064251 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad64afa0-477b-4d09-bb44-e50ed93e07a7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:00.067202 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad64afa0-477b-4d09-bb44-e50ed93e07a7 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:00.123072 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1209/2398] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:52:00 2022] POST /identity/v3/projects => generated 371 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:52:00.128885 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d278c838-b6a0-4467-9a33-339e25867acb None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:00.174970 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d278c838-b6a0-4467-9a33-339e25867acb None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:00.309597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242d1936-884d-422e-918e-b6f4fa0d762c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:00.310759 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242d1936-884d-422e-918e-b6f4fa0d762c admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:00.311252 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242d1936-884d-422e-918e-b6f4fa0d762c admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:00.312289 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242d1936-884d-422e-918e-b6f4fa0d762c admin admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:00.313572 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-242d1936-884d-422e-918e-b6f4fa0d762c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:00.317457 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-242d1936-884d-422e-918e-b6f4fa0d762c admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:00.354450 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1190/2399] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:52:00 2022] POST /identity/v3/users => generated 292 bytes in 226 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:52:00.359475 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad64afa0-477b-4d09-bb44-e50ed93e07a7 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:00.371996 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad64afa0-477b-4d09-bb44-e50ed93e07a7 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:00.379267 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31fe082-fc09-4a37-ad00-9ab535498022 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:00.379935 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31fe082-fc09-4a37-ad00-9ab535498022 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:00.380579 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31fe082-fc09-4a37-ad00-9ab535498022 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:00.381050 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31fe082-fc09-4a37-ad00-9ab535498022 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:00.381743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e31fe082-fc09-4a37-ad00-9ab535498022 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:00.383532 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e31fe082-fc09-4a37-ad00-9ab535498022 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:00.401031 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1210/2400] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:52:00 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:52:00.402811 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242d1936-884d-422e-918e-b6f4fa0d762c admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:00.423517 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242d1936-884d-422e-918e-b6f4fa0d762c admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:00.432531 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aee6217-26ea-433f-b8e4-510978fa7699 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:00.433717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aee6217-26ea-433f-b8e4-510978fa7699 admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:00.434213 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aee6217-26ea-433f-b8e4-510978fa7699 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:00.434768 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aee6217-26ea-433f-b8e4-510978fa7699 admin admin] PATH_INFO: `/v3/projects/09213a17ab164ade9dcf58d2399f1790/users/4de7d277bca0435b9eaafc9503fc9119/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:00.466577 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aee6217-26ea-433f-b8e4-510978fa7699 admin admin] RBAC: Authorizing `identity:create_grant(project_id=09213a17ab164ade9dcf58d2399f1790, user_id=4de7d277bca0435b9eaafc9503fc9119, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:00.474134 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aee6217-26ea-433f-b8e4-510978fa7699 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:00.504647 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1191/2401] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:52:00 2022] PUT /identity/v3/projects/09213a17ab164ade9dcf58d2399f1790/users/4de7d277bca0435b9eaafc9503fc9119/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:52:00.506152 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31fe082-fc09-4a37-ad00-9ab535498022 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:00.526663 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31fe082-fc09-4a37-ad00-9ab535498022 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:00.732936 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3fe7384-5986-41d4-8842-6512dff59012 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:00.735318 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fe7384-5986-41d4-8842-6512dff59012 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:00.736618 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fe7384-5986-41d4-8842-6512dff59012 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:00.737257 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fe7384-5986-41d4-8842-6512dff59012 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:00.739295 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3fe7384-5986-41d4-8842-6512dff59012 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:00.742796 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3fe7384-5986-41d4-8842-6512dff59012 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:00.801081 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1211/2402] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:52:00 2022] GET /identity/v3/roles => generated 1420 bytes in 295 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:52:00.806330 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aee6217-26ea-433f-b8e4-510978fa7699 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:00.829303 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aee6217-26ea-433f-b8e4-510978fa7699 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:00.863792 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d35dc8-ab9b-4c6f-abe5-1834382c714c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:00.865298 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d35dc8-ab9b-4c6f-abe5-1834382c714c admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:00.866327 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d35dc8-ab9b-4c6f-abe5-1834382c714c admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:00.867016 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d35dc8-ab9b-4c6f-abe5-1834382c714c admin admin] PATH_INFO: `/v3/projects/09213a17ab164ade9dcf58d2399f1790/users/4de7d277bca0435b9eaafc9503fc9119/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:00.872865 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d35dc8-ab9b-4c6f-abe5-1834382c714c admin admin] RBAC: Authorizing `identity:create_grant(project_id=09213a17ab164ade9dcf58d2399f1790, user_id=4de7d277bca0435b9eaafc9503fc9119, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:00.879069 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d35dc8-ab9b-4c6f-abe5-1834382c714c admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:00.913828 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1192/2403] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:52:00 2022] PUT /identity/v3/projects/09213a17ab164ade9dcf58d2399f1790/users/4de7d277bca0435b9eaafc9503fc9119/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:52:01.416301 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627a3268-b376-4af5-bac6-9708ca6eafb1 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:01.417090 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627a3268-b376-4af5-bac6-9708ca6eafb1 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:01.417090 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-627a3268-b376-4af5-bac6-9708ca6eafb1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:01.439059 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-627a3268-b376-4af5-bac6-9708ca6eafb1 None None] MFA Rules not processed for user `97c8877c74194a6bbbf25b3c8f5b7dbd`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:52:01.440348 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-627a3268-b376-4af5-bac6-9708ca6eafb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:01.547620 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1212/2404] 10.222.0.25 () {58 vars in 1009 bytes} [Fri Oct 28 09:52:01 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:52:01.555975 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d35dc8-ab9b-4c6f-abe5-1834382c714c admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:01.558557 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-19d35dc8-ab9b-4c6f-abe5-1834382c714c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:01.736526 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d35dc8-ab9b-4c6f-abe5-1834382c714c admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:01.755158 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b66c6f5e-8779-4d3e-a644-a12e0fd58adf service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c6cffe0d56e45c4b8fb54fbe5df6094', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:01.756124 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b66c6f5e-8779-4d3e-a644-a12e0fd58adf service swift] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:01.757892 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b66c6f5e-8779-4d3e-a644-a12e0fd58adf service swift] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:01.760497 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b66c6f5e-8779-4d3e-a644-a12e0fd58adf service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:01.788218 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b66c6f5e-8779-4d3e-a644-a12e0fd58adf service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:01.794295 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b66c6f5e-8779-4d3e-a644-a12e0fd58adf service swift] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:01.819508 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:52:01.844476 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1193/2405] 10.222.0.25 () {60 vars in 1384 bytes} [Fri Oct 28 09:52:01 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 289 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 28 09:52:11.533065 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=85,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:52:20.842760 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a1479f-ce19-414a-be8e-68324b4516be None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:20.843855 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a1479f-ce19-414a-be8e-68324b4516be None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:20.843855 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a1479f-ce19-414a-be8e-68324b4516be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:20.890341 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-f9a1479f-ce19-414a-be8e-68324b4516be None None] MFA Rules not processed for user `4de7d277bca0435b9eaafc9503fc9119`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:52:20.892172 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-f9a1479f-ce19-414a-be8e-68324b4516be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:20.904273 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b66c6f5e-8779-4d3e-a644-a12e0fd58adf service swift] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:20.909205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-b66c6f5e-8779-4d3e-a644-a12e0fd58adf service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:21.060710 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1213/2406] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:52:20 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:52:21.107259 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b66c6f5e-8779-4d3e-a644-a12e0fd58adf service swift] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:21.113155 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5cc9eb-177e-44eb-adfc-5ec4eb915005 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3f1d95049d2d4593bf52670b1a0035fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:21.114636 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5cc9eb-177e-44eb-adfc-5ec4eb915005 service placement] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:21.115260 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5cc9eb-177e-44eb-adfc-5ec4eb915005 service placement] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:21.115636 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5cc9eb-177e-44eb-adfc-5ec4eb915005 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:21.127997 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-9a5cc9eb-177e-44eb-adfc-5ec4eb915005 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:21.155469 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9a1479f-ce19-414a-be8e-68324b4516be None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:21.157388 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-f9a1479f-ce19-414a-be8e-68324b4516be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:21.167870 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a5cc9eb-177e-44eb-adfc-5ec4eb915005 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:21.174520 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a5cc9eb-177e-44eb-adfc-5ec4eb915005 service placement] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:21.198467 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1194/2407] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:52:20 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 295 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:52:21.293960 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5cc9eb-177e-44eb-adfc-5ec4eb915005 service placement] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:21.298231 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-9a5cc9eb-177e-44eb-adfc-5ec4eb915005 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:21.319287 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9a1479f-ce19-414a-be8e-68324b4516be None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:21.332186 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-017e3caf-7c2d-468f-bd83-0f5831fff54c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:21.336153 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017e3caf-7c2d-468f-bd83-0f5831fff54c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:21.337669 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017e3caf-7c2d-468f-bd83-0f5831fff54c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:21.338200 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017e3caf-7c2d-468f-bd83-0f5831fff54c admin admin] PATH_INFO: `/v3/users/a0e517bf245e44b99fd23325307ba2cb` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:21.345084 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-017e3caf-7c2d-468f-bd83-0f5831fff54c admin admin] RBAC: Authorizing `identity:delete_user(user_id=a0e517bf245e44b99fd23325307ba2cb)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:21.357876 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-017e3caf-7c2d-468f-bd83-0f5831fff54c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:21.505477 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-017e3caf-7c2d-468f-bd83-0f5831fff54c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a0e517bf245e44b99fd23325307ba2cb'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:21.532770 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-017e3caf-7c2d-468f-bd83-0f5831fff54c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a0e517bf245e44b99fd23325307ba2cb'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:21.534810 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-017e3caf-7c2d-468f-bd83-0f5831fff54c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a0e517bf245e44b99fd23325307ba2cb'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:21.539764 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-017e3caf-7c2d-468f-bd83-0f5831fff54c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a0e517bf245e44b99fd23325307ba2cb'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:21.552547 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5cc9eb-177e-44eb-adfc-5ec4eb915005 service placement] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:21.557034 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-017e3caf-7c2d-468f-bd83-0f5831fff54c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a0e517bf245e44b99fd23325307ba2cb'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:21.567035 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad4740c-d272-4d1a-8daf-c448a88ad5da service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:21.567978 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad4740c-d272-4d1a-8daf-c448a88ad5da service nova] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:21.568558 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad4740c-d272-4d1a-8daf-c448a88ad5da service nova] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:21.568989 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad4740c-d272-4d1a-8daf-c448a88ad5da service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:21.571735 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-dad4740c-d272-4d1a-8daf-c448a88ad5da service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:21.607995 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1214/2408] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:52:21 2022] DELETE /identity/v3/users/a0e517bf245e44b99fd23325307ba2cb => generated 0 bytes in 452 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:52:21.693602 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad4740c-d272-4d1a-8daf-c448a88ad5da service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:21.695742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad4740c-d272-4d1a-8daf-c448a88ad5da service nova] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:21.712810 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1195/2409] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:52:21 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 420 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:52:22.145045 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-017e3caf-7c2d-468f-bd83-0f5831fff54c admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:22.150028 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-017e3caf-7c2d-468f-bd83-0f5831fff54c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:22.162951 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:52:22.370981 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-017e3caf-7c2d-468f-bd83-0f5831fff54c admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:22.382049 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae9940af-3be7-4e4b-a331-6b586bf7b4e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:22.382991 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9940af-3be7-4e4b-a331-6b586bf7b4e2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:22.383270 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9940af-3be7-4e4b-a331-6b586bf7b4e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:22.383750 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9940af-3be7-4e4b-a331-6b586bf7b4e2 admin admin] PATH_INFO: `/v3/projects/6d071a4f24b541d4b5d49017707e0788` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:22.387298 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae9940af-3be7-4e4b-a331-6b586bf7b4e2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6d071a4f24b541d4b5d49017707e0788)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:22.389820 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae9940af-3be7-4e4b-a331-6b586bf7b4e2 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:22.515443 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ae9940af-3be7-4e4b-a331-6b586bf7b4e2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6d071a4f24b541d4b5d49017707e0788', 'request_id': 'req-ae9940af-3be7-4e4b-a331-6b586bf7b4e2'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:22.525266 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ae9940af-3be7-4e4b-a331-6b586bf7b4e2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6d071a4f24b541d4b5d49017707e0788', 'request_id': 'req-ae9940af-3be7-4e4b-a331-6b586bf7b4e2'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:22.540105 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ae9940af-3be7-4e4b-a331-6b586bf7b4e2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6d071a4f24b541d4b5d49017707e0788', 'request_id': 'req-ae9940af-3be7-4e4b-a331-6b586bf7b4e2'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:22.574131 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ae9940af-3be7-4e4b-a331-6b586bf7b4e2 admin admin] The token cache is being invalidate because project 6d071a4f24b541d4b5d49017707e0788 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:52:22.575374 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ae9940af-3be7-4e4b-a331-6b586bf7b4e2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:22.578063 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-ae9940af-3be7-4e4b-a331-6b586bf7b4e2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:22.582428 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1215/2410] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:52:22 2022] DELETE /identity/v3/projects/6d071a4f24b541d4b5d49017707e0788 => generated 0 bytes in 438 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:52:22.586790 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad4740c-d272-4d1a-8daf-c448a88ad5da service nova] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:22.608492 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad4740c-d272-4d1a-8daf-c448a88ad5da service nova] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:22.675144 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:52:22.758724 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a6f34c5-8ad2-478b-99bf-abbdbd66d173 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:22.758724 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6f34c5-8ad2-478b-99bf-abbdbd66d173 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:22.758724 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6f34c5-8ad2-478b-99bf-abbdbd66d173 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:22.758724 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6f34c5-8ad2-478b-99bf-abbdbd66d173 admin admin] PATH_INFO: `/v3/users/e48195948fb6421fbecd9bfd8aebe5da` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:22.759291 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a6f34c5-8ad2-478b-99bf-abbdbd66d173 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e48195948fb6421fbecd9bfd8aebe5da)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:22.759291 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a6f34c5-8ad2-478b-99bf-abbdbd66d173 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:22.883725 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-9a6f34c5-8ad2-478b-99bf-abbdbd66d173 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e48195948fb6421fbecd9bfd8aebe5da'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:22.898683 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-9a6f34c5-8ad2-478b-99bf-abbdbd66d173 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e48195948fb6421fbecd9bfd8aebe5da'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:22.900417 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-9a6f34c5-8ad2-478b-99bf-abbdbd66d173 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e48195948fb6421fbecd9bfd8aebe5da'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:22.904247 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-9a6f34c5-8ad2-478b-99bf-abbdbd66d173 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e48195948fb6421fbecd9bfd8aebe5da'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:22.922589 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-9a6f34c5-8ad2-478b-99bf-abbdbd66d173 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e48195948fb6421fbecd9bfd8aebe5da'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:23.020627 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1196/2411] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:52:22 2022] DELETE /identity/v3/users/e48195948fb6421fbecd9bfd8aebe5da => generated 0 bytes in 430 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:52:23.677850 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae9940af-3be7-4e4b-a331-6b586bf7b4e2 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:23.680514 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ae9940af-3be7-4e4b-a331-6b586bf7b4e2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:23.707994 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae9940af-3be7-4e4b-a331-6b586bf7b4e2 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:23.880348 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b019c3-88e1-4b0c-9807-ec2d98af85e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:23.881086 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b019c3-88e1-4b0c-9807-ec2d98af85e3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:23.881364 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b019c3-88e1-4b0c-9807-ec2d98af85e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:23.881646 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b019c3-88e1-4b0c-9807-ec2d98af85e3 admin admin] PATH_INFO: `/v3/projects/baf7f0be1a84414fb5e39b4d164db5f3` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:23.884884 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44b019c3-88e1-4b0c-9807-ec2d98af85e3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=baf7f0be1a84414fb5e39b4d164db5f3)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:23.886919 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44b019c3-88e1-4b0c-9807-ec2d98af85e3 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:23.976398 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:52:24.013128 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-44b019c3-88e1-4b0c-9807-ec2d98af85e3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'baf7f0be1a84414fb5e39b4d164db5f3', 'request_id': 'req-44b019c3-88e1-4b0c-9807-ec2d98af85e3'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:24.031421 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-44b019c3-88e1-4b0c-9807-ec2d98af85e3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'baf7f0be1a84414fb5e39b4d164db5f3', 'request_id': 'req-44b019c3-88e1-4b0c-9807-ec2d98af85e3'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:24.055762 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-44b019c3-88e1-4b0c-9807-ec2d98af85e3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'baf7f0be1a84414fb5e39b4d164db5f3', 'request_id': 'req-44b019c3-88e1-4b0c-9807-ec2d98af85e3'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:24.089680 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-44b019c3-88e1-4b0c-9807-ec2d98af85e3 admin admin] The token cache is being invalidate because project baf7f0be1a84414fb5e39b4d164db5f3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:52:24.090251 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-44b019c3-88e1-4b0c-9807-ec2d98af85e3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:24.091867 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-44b019c3-88e1-4b0c-9807-ec2d98af85e3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:24.110447 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1216/2412] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:52:23 2022] DELETE /identity/v3/projects/baf7f0be1a84414fb5e39b4d164db5f3 => generated 0 bytes in 419 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:52:24.831390 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e837f865-1434-4daf-8b19-a6eb956c2ce6 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:24.832873 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e837f865-1434-4daf-8b19-a6eb956c2ce6 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:24.833588 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e837f865-1434-4daf-8b19-a6eb956c2ce6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:24.888588 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-e837f865-1434-4daf-8b19-a6eb956c2ce6 None None] MFA Rules not processed for user `f14c3f34010b41ef9be2b9fd2aa98cd9`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:52:24.892892 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-e837f865-1434-4daf-8b19-a6eb956c2ce6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:25.004921 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1197/2413] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:52:24 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:52:26.497400 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b019c3-88e1-4b0c-9807-ec2d98af85e3 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:26.502744 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-44b019c3-88e1-4b0c-9807-ec2d98af85e3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:26.728807 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b019c3-88e1-4b0c-9807-ec2d98af85e3 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:26.734872 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c634181-5935-4f1f-b70c-8d8a46b05e0a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:26.742558 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c634181-5935-4f1f-b70c-8d8a46b05e0a service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:26.742558 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c634181-5935-4f1f-b70c-8d8a46b05e0a service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:26.742558 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c634181-5935-4f1f-b70c-8d8a46b05e0a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:26.760004 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-0c634181-5935-4f1f-b70c-8d8a46b05e0a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:26.928922 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c634181-5935-4f1f-b70c-8d8a46b05e0a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:26.931034 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c634181-5935-4f1f-b70c-8d8a46b05e0a service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:26.947419 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1217/2414] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:52:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 451 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:52:28.645166 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266b8a77-2d33-4034-9230-20321237b135 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:28.645822 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266b8a77-2d33-4034-9230-20321237b135 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:28.646206 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-266b8a77-2d33-4034-9230-20321237b135 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:28.683030 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-266b8a77-2d33-4034-9230-20321237b135 None None] MFA Rules not processed for user `97c8877c74194a6bbbf25b3c8f5b7dbd`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:52:28.685357 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-266b8a77-2d33-4034-9230-20321237b135 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:28.782507 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:52:28.807953 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1198/2415] 10.222.0.25 () {58 vars in 967 bytes} [Fri Oct 28 09:52:28 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:52:28.837503 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c634181-5935-4f1f-b70c-8d8a46b05e0a service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:28.839600 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-0c634181-5935-4f1f-b70c-8d8a46b05e0a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:28.956191 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c634181-5935-4f1f-b70c-8d8a46b05e0a service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:28.972094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b380b58-cd11-4811-a06d-c35e9db4e9b9 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c6cffe0d56e45c4b8fb54fbe5df6094', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:28.973010 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b380b58-cd11-4811-a06d-c35e9db4e9b9 service swift] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:28.973241 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b380b58-cd11-4811-a06d-c35e9db4e9b9 service swift] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:28.973781 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b380b58-cd11-4811-a06d-c35e9db4e9b9 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:29.009502 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b380b58-cd11-4811-a06d-c35e9db4e9b9 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:29.011699 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b380b58-cd11-4811-a06d-c35e9db4e9b9 service swift] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:29.039148 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1218/2416] 10.222.0.25 () {60 vars in 1384 bytes} [Fri Oct 28 09:52:28 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 202 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 28 09:52:32.413301 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-266b8a77-2d33-4034-9230-20321237b135 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:32.439010 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-266b8a77-2d33-4034-9230-20321237b135 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:32.448342 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a76c32b-8ecc-4c8d-8dce-0e394d409314 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:32.449174 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a76c32b-8ecc-4c8d-8dce-0e394d409314 service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:32.449616 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a76c32b-8ecc-4c8d-8dce-0e394d409314 service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:32.450025 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a76c32b-8ecc-4c8d-8dce-0e394d409314 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:32.452227 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-0a76c32b-8ecc-4c8d-8dce-0e394d409314 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:32.470692 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a76c32b-8ecc-4c8d-8dce-0e394d409314 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:32.472012 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a76c32b-8ecc-4c8d-8dce-0e394d409314 service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:32.506587 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1199/2417] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:52:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:52:32.731966 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b380b58-cd11-4811-a06d-c35e9db4e9b9 service swift] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:32.735655 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-8b380b58-cd11-4811-a06d-c35e9db4e9b9 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:32.754821 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:52:32.762934 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b380b58-cd11-4811-a06d-c35e9db4e9b9 service swift] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:32.772093 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f086e8-b5a5-4ada-8d50-46c95806b87f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:32.773306 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f086e8-b5a5-4ada-8d50-46c95806b87f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:32.773995 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f086e8-b5a5-4ada-8d50-46c95806b87f admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:32.774570 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f086e8-b5a5-4ada-8d50-46c95806b87f admin admin] PATH_INFO: `/v3/users/a4237ca76acd4f7483aaf66e1f5d2eea` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:32.777161 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5f086e8-b5a5-4ada-8d50-46c95806b87f admin admin] RBAC: Authorizing `identity:delete_user(user_id=a4237ca76acd4f7483aaf66e1f5d2eea)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:32.781687 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5f086e8-b5a5-4ada-8d50-46c95806b87f admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:32.876436 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a5f086e8-b5a5-4ada-8d50-46c95806b87f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a4237ca76acd4f7483aaf66e1f5d2eea'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:32.890024 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a5f086e8-b5a5-4ada-8d50-46c95806b87f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a4237ca76acd4f7483aaf66e1f5d2eea'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:32.891189 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a5f086e8-b5a5-4ada-8d50-46c95806b87f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a4237ca76acd4f7483aaf66e1f5d2eea'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:32.892232 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a5f086e8-b5a5-4ada-8d50-46c95806b87f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a4237ca76acd4f7483aaf66e1f5d2eea'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:32.908483 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a5f086e8-b5a5-4ada-8d50-46c95806b87f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a4237ca76acd4f7483aaf66e1f5d2eea'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:32.946613 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1219/2418] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:52:32 2022] DELETE /identity/v3/users/a4237ca76acd4f7483aaf66e1f5d2eea => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:52:33.394854 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a76c32b-8ecc-4c8d-8dce-0e394d409314 service neutron] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:33.410086 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a76c32b-8ecc-4c8d-8dce-0e394d409314 service neutron] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:33.528554 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b745c28f-358d-443e-bb16-d562ca565b94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:33.529890 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b745c28f-358d-443e-bb16-d562ca565b94 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:33.530381 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b745c28f-358d-443e-bb16-d562ca565b94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:33.530923 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b745c28f-358d-443e-bb16-d562ca565b94 admin admin] PATH_INFO: `/v3/projects/f2fcce993e6b4388bde493f0702fd49d` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:33.533119 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b745c28f-358d-443e-bb16-d562ca565b94 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f2fcce993e6b4388bde493f0702fd49d)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:33.537300 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b745c28f-358d-443e-bb16-d562ca565b94 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:33.647254 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-b745c28f-358d-443e-bb16-d562ca565b94 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f2fcce993e6b4388bde493f0702fd49d', 'request_id': 'req-b745c28f-358d-443e-bb16-d562ca565b94'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:33.663673 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-b745c28f-358d-443e-bb16-d562ca565b94 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f2fcce993e6b4388bde493f0702fd49d', 'request_id': 'req-b745c28f-358d-443e-bb16-d562ca565b94'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:33.678522 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-b745c28f-358d-443e-bb16-d562ca565b94 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f2fcce993e6b4388bde493f0702fd49d', 'request_id': 'req-b745c28f-358d-443e-bb16-d562ca565b94'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:33.699433 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-b745c28f-358d-443e-bb16-d562ca565b94 admin admin] The token cache is being invalidate because project f2fcce993e6b4388bde493f0702fd49d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:52:33.699859 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-b745c28f-358d-443e-bb16-d562ca565b94 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:33.701035 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-b745c28f-358d-443e-bb16-d562ca565b94 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:33.703468 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1200/2419] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:52:33 2022] DELETE /identity/v3/projects/f2fcce993e6b4388bde493f0702fd49d => generated 0 bytes in 309 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:52:33.708907 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a0dd66-2f3c-4146-acf2-46bbe71e08ad None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:33.709202 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a0dd66-2f3c-4146-acf2-46bbe71e08ad None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:33.709485 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a0dd66-2f3c-4146-acf2-46bbe71e08ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:33.739543 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-30a0dd66-2f3c-4146-acf2-46bbe71e08ad None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:52:33.740901 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-30a0dd66-2f3c-4146-acf2-46bbe71e08ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:33.839315 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1220/2420] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:52:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:52:33.849196 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e045b16-83ef-4a68-83b5-e2723f27bac8 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:33.849484 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e045b16-83ef-4a68-83b5-e2723f27bac8 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:33.849764 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e045b16-83ef-4a68-83b5-e2723f27bac8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:34.012615 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-0e045b16-83ef-4a68-83b5-e2723f27bac8 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:52:34.014032 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-0e045b16-83ef-4a68-83b5-e2723f27bac8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:34.026180 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1201/2421] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:52:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:52:34.033330 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a0dd66-2f3c-4146-acf2-46bbe71e08ad None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:34.066484 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a0dd66-2f3c-4146-acf2-46bbe71e08ad None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:34.075596 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-393d2764-5e78-465b-8296-e4e28446e7ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:34.079008 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393d2764-5e78-465b-8296-e4e28446e7ce admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:34.079622 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393d2764-5e78-465b-8296-e4e28446e7ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:34.080118 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393d2764-5e78-465b-8296-e4e28446e7ce admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:34.081374 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-393d2764-5e78-465b-8296-e4e28446e7ce admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:52:34.081920 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-393d2764-5e78-465b-8296-e4e28446e7ce admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:34.085155 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-393d2764-5e78-465b-8296-e4e28446e7ce admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:34.111250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1221/2422] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:52:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:52:34.118356 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e045b16-83ef-4a68-83b5-e2723f27bac8 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:34.131394 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e045b16-83ef-4a68-83b5-e2723f27bac8 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:34.137467 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c61ea52-5da1-46fc-adba-80775d9373ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:34.138319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c61ea52-5da1-46fc-adba-80775d9373ec admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:34.138685 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c61ea52-5da1-46fc-adba-80775d9373ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:34.139083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c61ea52-5da1-46fc-adba-80775d9373ec admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:34.139921 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c61ea52-5da1-46fc-adba-80775d9373ec admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:34.142839 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c61ea52-5da1-46fc-adba-80775d9373ec admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:34.198954 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1202/2423] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:52:34 2022] POST /identity/v3/projects => generated 403 bytes in 81 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:52:34.205493 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-393d2764-5e78-465b-8296-e4e28446e7ce admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:34.220519 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-393d2764-5e78-465b-8296-e4e28446e7ce admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:34.339200 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd421a2a-f6a7-4aa4-95fb-f9459b655571 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:34.340174 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd421a2a-f6a7-4aa4-95fb-f9459b655571 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:34.340458 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd421a2a-f6a7-4aa4-95fb-f9459b655571 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:34.340943 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd421a2a-f6a7-4aa4-95fb-f9459b655571 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:34.342058 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd421a2a-f6a7-4aa4-95fb-f9459b655571 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:34.345798 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd421a2a-f6a7-4aa4-95fb-f9459b655571 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:34.361412 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=87,keystone:DELETE=10,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:52:34.378799 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1222/2424] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:52:34 2022] POST /identity/v3/users => generated 302 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:52:34.383851 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c61ea52-5da1-46fc-adba-80775d9373ec admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:34.404116 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c61ea52-5da1-46fc-adba-80775d9373ec admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:34.413313 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e14e0cad-1caf-4168-860f-5e0f1f7a939d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:34.414434 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e14e0cad-1caf-4168-860f-5e0f1f7a939d admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:34.415058 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e14e0cad-1caf-4168-860f-5e0f1f7a939d admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:34.415659 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e14e0cad-1caf-4168-860f-5e0f1f7a939d admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:34.416942 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e14e0cad-1caf-4168-860f-5e0f1f7a939d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:34.420147 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e14e0cad-1caf-4168-860f-5e0f1f7a939d admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:34.436964 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1203/2425] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:52:34 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:52:34.441567 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd421a2a-f6a7-4aa4-95fb-f9459b655571 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:34.453324 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd421a2a-f6a7-4aa4-95fb-f9459b655571 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:34.458386 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b07ec16-1a15-426c-93ed-06407fd34fb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:34.459057 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b07ec16-1a15-426c-93ed-06407fd34fb7 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:34.459339 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b07ec16-1a15-426c-93ed-06407fd34fb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:34.459610 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b07ec16-1a15-426c-93ed-06407fd34fb7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:34.460544 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b07ec16-1a15-426c-93ed-06407fd34fb7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:34.462048 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b07ec16-1a15-426c-93ed-06407fd34fb7 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:34.480996 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1223/2426] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:52:34 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:52:34.482340 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e14e0cad-1caf-4168-860f-5e0f1f7a939d admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:34.498768 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e14e0cad-1caf-4168-860f-5e0f1f7a939d admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:34.513467 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-528da48d-00a6-4fa6-83e1-796efe5d4e44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:34.515735 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528da48d-00a6-4fa6-83e1-796efe5d4e44 admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:34.516108 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528da48d-00a6-4fa6-83e1-796efe5d4e44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:34.516449 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528da48d-00a6-4fa6-83e1-796efe5d4e44 admin admin] PATH_INFO: `/v3/projects/e55674d8a23540c5bed9195c78591793/users/691b7078a27e4605be893387942425bc/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:34.535762 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-528da48d-00a6-4fa6-83e1-796efe5d4e44 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e55674d8a23540c5bed9195c78591793, user_id=691b7078a27e4605be893387942425bc, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:34.539176 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-528da48d-00a6-4fa6-83e1-796efe5d4e44 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:34.561875 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1204/2427] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:52:34 2022] PUT /identity/v3/projects/e55674d8a23540c5bed9195c78591793/users/691b7078a27e4605be893387942425bc/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 80 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:52:34.568843 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644b3cb5-eb4a-4489-9802-141f889341fc None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:34.569177 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644b3cb5-eb4a-4489-9802-141f889341fc None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:34.569746 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644b3cb5-eb4a-4489-9802-141f889341fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:34.618591 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-644b3cb5-eb4a-4489-9802-141f889341fc None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:52:34.621539 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-644b3cb5-eb4a-4489-9802-141f889341fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:34.711602 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1224/2428] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:52:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:52:34.726063 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-528da48d-00a6-4fa6-83e1-796efe5d4e44 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:34.730339 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-528da48d-00a6-4fa6-83e1-796efe5d4e44 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:34.757382 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:52:34.859905 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-528da48d-00a6-4fa6-83e1-796efe5d4e44 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:34.866941 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04fada4-8720-4f3a-8ec5-34d1d74e6fbb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:34.868007 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04fada4-8720-4f3a-8ec5-34d1d74e6fbb service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:34.868518 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04fada4-8720-4f3a-8ec5-34d1d74e6fbb service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:34.869077 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04fada4-8720-4f3a-8ec5-34d1d74e6fbb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:34.885944 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04fada4-8720-4f3a-8ec5-34d1d74e6fbb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:34.887759 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04fada4-8720-4f3a-8ec5-34d1d74e6fbb service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:34.900683 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1205/2429] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:52:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:52:36.051460 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc46609-f998-43f5-9309-6abbfbf9fbd2 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:36.052384 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc46609-f998-43f5-9309-6abbfbf9fbd2 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:36.052661 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc46609-f998-43f5-9309-6abbfbf9fbd2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:36.111937 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-9fc46609-f998-43f5-9309-6abbfbf9fbd2 None None] MFA Rules not processed for user `691b7078a27e4605be893387942425bc`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:52:36.114084 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-9fc46609-f998-43f5-9309-6abbfbf9fbd2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:36.290785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1225/2430] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:52:36 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 241 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:52:36.300035 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04fada4-8720-4f3a-8ec5-34d1d74e6fbb service neutron] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:36.341773 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04fada4-8720-4f3a-8ec5-34d1d74e6fbb service neutron] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:36.355945 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cfa39c-8cd4-4adb-91eb-bb650a3fbb24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:36.357014 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cfa39c-8cd4-4adb-91eb-bb650a3fbb24 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:36.357290 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cfa39c-8cd4-4adb-91eb-bb650a3fbb24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:36.359778 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72cfa39c-8cd4-4adb-91eb-bb650a3fbb24 admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:36.360720 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72cfa39c-8cd4-4adb-91eb-bb650a3fbb24 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:36.363112 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72cfa39c-8cd4-4adb-91eb-bb650a3fbb24 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:36.408003 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1206/2431] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:52:36 2022] POST /identity/v3/projects => generated 405 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:52:36.413617 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc46609-f998-43f5-9309-6abbfbf9fbd2 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:36.429094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc46609-f998-43f5-9309-6abbfbf9fbd2 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:36.538273 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94bfaef-f8ef-46ea-b625-c206a9708503 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:36.538273 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94bfaef-f8ef-46ea-b625-c206a9708503 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:36.538673 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94bfaef-f8ef-46ea-b625-c206a9708503 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:36.539195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94bfaef-f8ef-46ea-b625-c206a9708503 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:36.540465 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94bfaef-f8ef-46ea-b625-c206a9708503 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:36.544004 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94bfaef-f8ef-46ea-b625-c206a9708503 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:36.589954 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1226/2432] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:52:36 2022] POST /identity/v3/users => generated 309 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:52:36.591103 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cfa39c-8cd4-4adb-91eb-bb650a3fbb24 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:36.607004 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72cfa39c-8cd4-4adb-91eb-bb650a3fbb24 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:36.616054 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d109894-e1c6-44a9-baec-513861bc4171 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:36.617548 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d109894-e1c6-44a9-baec-513861bc4171 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:36.618124 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d109894-e1c6-44a9-baec-513861bc4171 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:36.618702 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d109894-e1c6-44a9-baec-513861bc4171 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:36.620068 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d109894-e1c6-44a9-baec-513861bc4171 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:36.623087 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d109894-e1c6-44a9-baec-513861bc4171 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:36.647615 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1207/2433] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:52:36 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:52:36.654177 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94bfaef-f8ef-46ea-b625-c206a9708503 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:36.666447 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:INSERT=4,keystone:SELECT=96 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:52:36.681893 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94bfaef-f8ef-46ea-b625-c206a9708503 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:36.690979 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4a8b47-ecc3-44c7-9d6b-4909cb92dbb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:36.693516 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4a8b47-ecc3-44c7-9d6b-4909cb92dbb5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:36.694013 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4a8b47-ecc3-44c7-9d6b-4909cb92dbb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:36.695766 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4a8b47-ecc3-44c7-9d6b-4909cb92dbb5 admin admin] PATH_INFO: `/v3/projects/57d28e1bf8b247f8887df8135d92a40b/users/7d7cb2c1ddef474698c8c03ac8b09213/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:36.729194 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e4a8b47-ecc3-44c7-9d6b-4909cb92dbb5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=57d28e1bf8b247f8887df8135d92a40b, user_id=7d7cb2c1ddef474698c8c03ac8b09213, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:36.736605 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e4a8b47-ecc3-44c7-9d6b-4909cb92dbb5 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:36.777642 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1227/2434] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:52:36 2022] PUT /identity/v3/projects/57d28e1bf8b247f8887df8135d92a40b/users/7d7cb2c1ddef474698c8c03ac8b09213/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:52:36.782656 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d109894-e1c6-44a9-baec-513861bc4171 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:36.809669 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d109894-e1c6-44a9-baec-513861bc4171 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:36.979093 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6e2204-6bc7-4342-ba5d-bada40a5d8f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:36.979093 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6e2204-6bc7-4342-ba5d-bada40a5d8f7 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:36.979093 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6e2204-6bc7-4342-ba5d-bada40a5d8f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:36.979093 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6e2204-6bc7-4342-ba5d-bada40a5d8f7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:36.979828 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e6e2204-6bc7-4342-ba5d-bada40a5d8f7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:36.979922 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e6e2204-6bc7-4342-ba5d-bada40a5d8f7 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:37.010664 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1208/2435] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:52:36 2022] GET /identity/v3/roles => generated 1420 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:52:37.012141 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4a8b47-ecc3-44c7-9d6b-4909cb92dbb5 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:37.035422 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4a8b47-ecc3-44c7-9d6b-4909cb92dbb5 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:37.044720 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff4491f-9136-4db2-a199-8c4a378f1b3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:37.046838 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff4491f-9136-4db2-a199-8c4a378f1b3a admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:37.047508 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff4491f-9136-4db2-a199-8c4a378f1b3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:37.048081 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff4491f-9136-4db2-a199-8c4a378f1b3a admin admin] PATH_INFO: `/v3/projects/57d28e1bf8b247f8887df8135d92a40b/users/7d7cb2c1ddef474698c8c03ac8b09213/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:37.052425 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff4491f-9136-4db2-a199-8c4a378f1b3a admin admin] RBAC: Authorizing `identity:create_grant(project_id=57d28e1bf8b247f8887df8135d92a40b, user_id=7d7cb2c1ddef474698c8c03ac8b09213, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:37.058096 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff4491f-9136-4db2-a199-8c4a378f1b3a admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:37.081101 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1228/2436] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:52:37 2022] PUT /identity/v3/projects/57d28e1bf8b247f8887df8135d92a40b/users/7d7cb2c1ddef474698c8c03ac8b09213/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:52:37.875256 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6e2204-6bc7-4342-ba5d-bada40a5d8f7 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:37.878645 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-8e6e2204-6bc7-4342-ba5d-bada40a5d8f7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:37.897106 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6e2204-6bc7-4342-ba5d-bada40a5d8f7 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:37.990440 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb87e80-c220-46cc-8a9f-f5be9ebc4c1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:37.991483 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb87e80-c220-46cc-8a9f-f5be9ebc4c1d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:37.992130 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb87e80-c220-46cc-8a9f-f5be9ebc4c1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:37.992643 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb87e80-c220-46cc-8a9f-f5be9ebc4c1d admin admin] PATH_INFO: `/v3/users/e65d298cca0f478eba502edf5bd13954` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:37.995416 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eb87e80-c220-46cc-8a9f-f5be9ebc4c1d admin admin] RBAC: Authorizing `identity:delete_user(user_id=e65d298cca0f478eba502edf5bd13954)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:38.000481 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eb87e80-c220-46cc-8a9f-f5be9ebc4c1d admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:38.034637 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=98,keystone:INSERT=1,keystone:DELETE=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:52:38.097897 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-6eb87e80-c220-46cc-8a9f-f5be9ebc4c1d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e65d298cca0f478eba502edf5bd13954'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:38.107961 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-6eb87e80-c220-46cc-8a9f-f5be9ebc4c1d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e65d298cca0f478eba502edf5bd13954'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:38.108784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-6eb87e80-c220-46cc-8a9f-f5be9ebc4c1d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e65d298cca0f478eba502edf5bd13954'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:38.109652 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-6eb87e80-c220-46cc-8a9f-f5be9ebc4c1d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e65d298cca0f478eba502edf5bd13954'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:38.122527 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-6eb87e80-c220-46cc-8a9f-f5be9ebc4c1d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e65d298cca0f478eba502edf5bd13954'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:38.149932 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1209/2437] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:52:37 2022] DELETE /identity/v3/users/e65d298cca0f478eba502edf5bd13954 => generated 0 bytes in 275 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:52:38.300411 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad83b0bb-ae1a-4482-a41c-74706d2489af None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:38.301049 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad83b0bb-ae1a-4482-a41c-74706d2489af None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:38.301449 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad83b0bb-ae1a-4482-a41c-74706d2489af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:38.357354 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-ad83b0bb-ae1a-4482-a41c-74706d2489af None None] MFA Rules not processed for user `7d7cb2c1ddef474698c8c03ac8b09213`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:52:38.359768 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ad83b0bb-ae1a-4482-a41c-74706d2489af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:38.445374 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb87e80-c220-46cc-8a9f-f5be9ebc4c1d admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:38.449295 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-6eb87e80-c220-46cc-8a9f-f5be9ebc4c1d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:38.537613 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1229/2438] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:52:38 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 239 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:52:38.584893 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb87e80-c220-46cc-8a9f-f5be9ebc4c1d admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:38.599852 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc16e41-3f71-4496-b990-326a9c4652ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:38.601255 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc16e41-3f71-4496-b990-326a9c4652ab admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:38.602708 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc16e41-3f71-4496-b990-326a9c4652ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:38.602708 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc16e41-3f71-4496-b990-326a9c4652ab admin admin] PATH_INFO: `/v3/projects/4f351d016ace4e369be852bb7aa916fe` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:38.604678 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cc16e41-3f71-4496-b990-326a9c4652ab admin admin] RBAC: Authorizing `identity:delete_project(project_id=4f351d016ace4e369be852bb7aa916fe)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:38.608440 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cc16e41-3f71-4496-b990-326a9c4652ab admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:38.711725 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-3cc16e41-3f71-4496-b990-326a9c4652ab admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4f351d016ace4e369be852bb7aa916fe', 'request_id': 'req-3cc16e41-3f71-4496-b990-326a9c4652ab'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:38.726568 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-3cc16e41-3f71-4496-b990-326a9c4652ab admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4f351d016ace4e369be852bb7aa916fe', 'request_id': 'req-3cc16e41-3f71-4496-b990-326a9c4652ab'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:38.743595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-3cc16e41-3f71-4496-b990-326a9c4652ab admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4f351d016ace4e369be852bb7aa916fe', 'request_id': 'req-3cc16e41-3f71-4496-b990-326a9c4652ab'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:38.768614 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-3cc16e41-3f71-4496-b990-326a9c4652ab admin admin] The token cache is being invalidate because project 4f351d016ace4e369be852bb7aa916fe was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:52:38.769141 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-3cc16e41-3f71-4496-b990-326a9c4652ab admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:38.770295 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-3cc16e41-3f71-4496-b990-326a9c4652ab admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:38.773082 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1210/2439] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:52:38 2022] DELETE /identity/v3/projects/4f351d016ace4e369be852bb7aa916fe => generated 0 bytes in 330 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:52:38.780506 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e065a5c1-c3ce-45f8-8327-329ba01393e0 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:38.780951 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e065a5c1-c3ce-45f8-8327-329ba01393e0 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:38.781444 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e065a5c1-c3ce-45f8-8327-329ba01393e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:38.813675 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-e065a5c1-c3ce-45f8-8327-329ba01393e0 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:52:38.816345 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e065a5c1-c3ce-45f8-8327-329ba01393e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:38.928056 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1230/2440] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:52:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:52:38.933367 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb873fd-563f-453c-bc62-503bec2b61a9 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:38.934137 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb873fd-563f-453c-bc62-503bec2b61a9 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:38.934734 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb873fd-563f-453c-bc62-503bec2b61a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:38.970123 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-edb873fd-563f-453c-bc62-503bec2b61a9 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:52:38.971561 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-edb873fd-563f-453c-bc62-503bec2b61a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:38.986769 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1211/2441] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:52:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 56 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:52:38.995315 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e065a5c1-c3ce-45f8-8327-329ba01393e0 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:39.016248 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e065a5c1-c3ce-45f8-8327-329ba01393e0 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:39.026439 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e206f27-16e9-40c6-803e-db2a34dfbafa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:39.027596 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e206f27-16e9-40c6-803e-db2a34dfbafa admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:39.028169 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e206f27-16e9-40c6-803e-db2a34dfbafa admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:39.028739 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e206f27-16e9-40c6-803e-db2a34dfbafa admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:39.030178 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e206f27-16e9-40c6-803e-db2a34dfbafa admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:52:39.030842 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e206f27-16e9-40c6-803e-db2a34dfbafa admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:39.034250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e206f27-16e9-40c6-803e-db2a34dfbafa admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:39.061667 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1231/2442] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:52:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:52:39.065425 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb873fd-563f-453c-bc62-503bec2b61a9 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:39.078807 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb873fd-563f-453c-bc62-503bec2b61a9 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:39.091308 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914ee0a4-5842-4e3d-aa2d-0664d297457a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:39.092834 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914ee0a4-5842-4e3d-aa2d-0664d297457a admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:39.093675 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914ee0a4-5842-4e3d-aa2d-0664d297457a admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:39.094705 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914ee0a4-5842-4e3d-aa2d-0664d297457a admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:39.096324 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914ee0a4-5842-4e3d-aa2d-0664d297457a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:39.104110 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-914ee0a4-5842-4e3d-aa2d-0664d297457a admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:39.134214 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:DELETE=16,keystone:SELECT=81,keystone:INSERT=3 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:52:39.166452 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1212/2443] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:52:39 2022] POST /identity/v3/projects => generated 373 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:52:39.170571 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e206f27-16e9-40c6-803e-db2a34dfbafa admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:39.192013 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e206f27-16e9-40c6-803e-db2a34dfbafa admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:39.255762 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:52:39.291176 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd2e054-62e8-4462-934e-f3972aa03e47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:39.291983 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd2e054-62e8-4462-934e-f3972aa03e47 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:39.292369 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd2e054-62e8-4462-934e-f3972aa03e47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:39.292762 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd2e054-62e8-4462-934e-f3972aa03e47 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:39.293709 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd2e054-62e8-4462-934e-f3972aa03e47 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:39.297845 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd2e054-62e8-4462-934e-f3972aa03e47 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:39.331105 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1232/2444] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:52:39 2022] POST /identity/v3/users => generated 287 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:52:39.337405 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914ee0a4-5842-4e3d-aa2d-0664d297457a admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:39.354972 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914ee0a4-5842-4e3d-aa2d-0664d297457a admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:39.362170 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430a5db5-3495-49db-8bbc-169be9b8ee9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:39.363253 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430a5db5-3495-49db-8bbc-169be9b8ee9c admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:39.363870 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430a5db5-3495-49db-8bbc-169be9b8ee9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:39.364238 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430a5db5-3495-49db-8bbc-169be9b8ee9c admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:39.365435 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-430a5db5-3495-49db-8bbc-169be9b8ee9c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:39.369098 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-430a5db5-3495-49db-8bbc-169be9b8ee9c admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:39.391319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1213/2445] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:52:39 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:52:39.396152 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd2e054-62e8-4462-934e-f3972aa03e47 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:39.414361 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd2e054-62e8-4462-934e-f3972aa03e47 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:39.424402 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1d3759-9fa7-4a33-891f-76157d65873c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:39.425494 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1d3759-9fa7-4a33-891f-76157d65873c admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:39.426104 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1d3759-9fa7-4a33-891f-76157d65873c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:39.426747 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1d3759-9fa7-4a33-891f-76157d65873c admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:39.428466 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a1d3759-9fa7-4a33-891f-76157d65873c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:39.432526 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a1d3759-9fa7-4a33-891f-76157d65873c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:39.458245 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1233/2446] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:52:39 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:52:39.463185 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430a5db5-3495-49db-8bbc-169be9b8ee9c admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:39.475920 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430a5db5-3495-49db-8bbc-169be9b8ee9c admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:39.483335 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-359e0a86-60b1-4b37-bd64-aed923316177 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:39.484205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359e0a86-60b1-4b37-bd64-aed923316177 admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:39.484561 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359e0a86-60b1-4b37-bd64-aed923316177 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:39.484887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359e0a86-60b1-4b37-bd64-aed923316177 admin admin] PATH_INFO: `/v3/projects/cc0b8c4858ad4ccbb73c54cd7012d924/users/1ace0b58f73c4279adf024cc677330bd/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:39.506828 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-359e0a86-60b1-4b37-bd64-aed923316177 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cc0b8c4858ad4ccbb73c54cd7012d924, user_id=1ace0b58f73c4279adf024cc677330bd, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:39.509723 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-359e0a86-60b1-4b37-bd64-aed923316177 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:39.526166 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1214/2447] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:52:39 2022] PUT /identity/v3/projects/cc0b8c4858ad4ccbb73c54cd7012d924/users/1ace0b58f73c4279adf024cc677330bd/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:52:39.539879 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa010ea-f208-424c-95bb-1d935db6b51e None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:39.540198 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa010ea-f208-424c-95bb-1d935db6b51e None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:39.540436 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa010ea-f208-424c-95bb-1d935db6b51e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:39.572555 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-1aa010ea-f208-424c-95bb-1d935db6b51e None None] MFA Rules not processed for user `1ace0b58f73c4279adf024cc677330bd`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:52:39.573805 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-1aa010ea-f208-424c-95bb-1d935db6b51e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:39.681518 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1234/2448] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:52:39 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:52:39.695206 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-359e0a86-60b1-4b37-bd64-aed923316177 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:39.698501 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-359e0a86-60b1-4b37-bd64-aed923316177 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:39.787924 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-359e0a86-60b1-4b37-bd64-aed923316177 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:39.794138 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb13b9a8-09f7-4996-aca5-4d693c27149a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:39.794950 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb13b9a8-09f7-4996-aca5-4d693c27149a service nova] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:39.795328 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb13b9a8-09f7-4996-aca5-4d693c27149a service nova] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:39.795602 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb13b9a8-09f7-4996-aca5-4d693c27149a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:39.805063 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb13b9a8-09f7-4996-aca5-4d693c27149a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:39.806750 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb13b9a8-09f7-4996-aca5-4d693c27149a service nova] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:39.817834 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1215/2449] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:52:39 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:52:40.197584 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa010ea-f208-424c-95bb-1d935db6b51e None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:40.202230 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-1aa010ea-f208-424c-95bb-1d935db6b51e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:40.362962 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa010ea-f208-424c-95bb-1d935db6b51e None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:40.370071 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3713be-2923-481b-93d8-7a317f28ec64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:40.370946 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3713be-2923-481b-93d8-7a317f28ec64 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:40.371292 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3713be-2923-481b-93d8-7a317f28ec64 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:40.371717 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b3713be-2923-481b-93d8-7a317f28ec64 admin admin] PATH_INFO: `/v3/users/691b7078a27e4605be893387942425bc` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:40.374113 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b3713be-2923-481b-93d8-7a317f28ec64 admin admin] RBAC: Authorizing `identity:delete_user(user_id=691b7078a27e4605be893387942425bc)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:40.376205 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b3713be-2923-481b-93d8-7a317f28ec64 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:40.453955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-2b3713be-2923-481b-93d8-7a317f28ec64 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '691b7078a27e4605be893387942425bc'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:40.455707 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=90,keystone:INSERT=3,keystone:DELETE=7 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:52:40.464336 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-2b3713be-2923-481b-93d8-7a317f28ec64 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '691b7078a27e4605be893387942425bc'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:40.465268 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-2b3713be-2923-481b-93d8-7a317f28ec64 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '691b7078a27e4605be893387942425bc'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:40.466198 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-2b3713be-2923-481b-93d8-7a317f28ec64 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '691b7078a27e4605be893387942425bc'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:40.476174 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-2b3713be-2923-481b-93d8-7a317f28ec64 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '691b7078a27e4605be893387942425bc'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:40.498074 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1235/2450] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:52:40 2022] DELETE /identity/v3/users/691b7078a27e4605be893387942425bc => generated 0 bytes in 302 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:52:40.692329 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb13b9a8-09f7-4996-aca5-4d693c27149a service nova] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:40.711909 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb13b9a8-09f7-4996-aca5-4d693c27149a service nova] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:40.790821 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f8bc05-6d21-40c3-a173-869cba50e639 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:40.791586 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f8bc05-6d21-40c3-a173-869cba50e639 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:40.791914 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f8bc05-6d21-40c3-a173-869cba50e639 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:40.792292 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f8bc05-6d21-40c3-a173-869cba50e639 admin admin] PATH_INFO: `/v3/projects/e55674d8a23540c5bed9195c78591793` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:40.793613 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36f8bc05-6d21-40c3-a173-869cba50e639 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e55674d8a23540c5bed9195c78591793)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:40.796307 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36f8bc05-6d21-40c3-a173-869cba50e639 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:40.854208 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=95,keystone:INSERT=1,keystone:DELETE=4 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:52:40.856490 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-36f8bc05-6d21-40c3-a173-869cba50e639 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e55674d8a23540c5bed9195c78591793', 'request_id': 'req-36f8bc05-6d21-40c3-a173-869cba50e639'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:40.867452 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-36f8bc05-6d21-40c3-a173-869cba50e639 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e55674d8a23540c5bed9195c78591793', 'request_id': 'req-36f8bc05-6d21-40c3-a173-869cba50e639'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:40.877632 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-36f8bc05-6d21-40c3-a173-869cba50e639 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e55674d8a23540c5bed9195c78591793', 'request_id': 'req-36f8bc05-6d21-40c3-a173-869cba50e639'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:40.889680 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-36f8bc05-6d21-40c3-a173-869cba50e639 admin admin] The token cache is being invalidate because project e55674d8a23540c5bed9195c78591793 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:52:40.889984 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-36f8bc05-6d21-40c3-a173-869cba50e639 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:40.890742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-36f8bc05-6d21-40c3-a173-869cba50e639 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:40.892272 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1216/2451] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:52:40 2022] DELETE /identity/v3/projects/e55674d8a23540c5bed9195c78591793 => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:52:40.897132 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3713be-2923-481b-93d8-7a317f28ec64 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:40.898899 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-2b3713be-2923-481b-93d8-7a317f28ec64 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:40.919594 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b3713be-2923-481b-93d8-7a317f28ec64 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:40.995584 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c65e9656-9828-4872-a3f3-3f3034118747 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:40.995584 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65e9656-9828-4872-a3f3-3f3034118747 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:40.995584 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65e9656-9828-4872-a3f3-3f3034118747 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:40.996959 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c65e9656-9828-4872-a3f3-3f3034118747 admin admin] PATH_INFO: `/v3/users/7d7cb2c1ddef474698c8c03ac8b09213` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:40.997935 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c65e9656-9828-4872-a3f3-3f3034118747 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7d7cb2c1ddef474698c8c03ac8b09213)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:40.999960 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c65e9656-9828-4872-a3f3-3f3034118747 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:41.057178 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c65e9656-9828-4872-a3f3-3f3034118747 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7d7cb2c1ddef474698c8c03ac8b09213'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:41.068552 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c65e9656-9828-4872-a3f3-3f3034118747 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7d7cb2c1ddef474698c8c03ac8b09213'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:41.069831 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c65e9656-9828-4872-a3f3-3f3034118747 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7d7cb2c1ddef474698c8c03ac8b09213'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:41.069831 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c65e9656-9828-4872-a3f3-3f3034118747 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7d7cb2c1ddef474698c8c03ac8b09213'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:41.079244 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c65e9656-9828-4872-a3f3-3f3034118747 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7d7cb2c1ddef474698c8c03ac8b09213'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:41.099830 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1236/2452] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:52:40 2022] DELETE /identity/v3/users/7d7cb2c1ddef474698c8c03ac8b09213 => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:52:41.331369 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f8bc05-6d21-40c3-a173-869cba50e639 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:41.333181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-36f8bc05-6d21-40c3-a173-869cba50e639 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:41.353571 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f8bc05-6d21-40c3-a173-869cba50e639 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:41.439593 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3bdd303-8396-4a75-ab82-0ded96a26661 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:41.440333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3bdd303-8396-4a75-ab82-0ded96a26661 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:41.440691 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3bdd303-8396-4a75-ab82-0ded96a26661 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:41.441043 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3bdd303-8396-4a75-ab82-0ded96a26661 admin admin] PATH_INFO: `/v3/projects/57d28e1bf8b247f8887df8135d92a40b` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:41.442425 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3bdd303-8396-4a75-ab82-0ded96a26661 admin admin] RBAC: Authorizing `identity:delete_project(project_id=57d28e1bf8b247f8887df8135d92a40b)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:41.444413 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3bdd303-8396-4a75-ab82-0ded96a26661 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:41.507691 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-a3bdd303-8396-4a75-ab82-0ded96a26661 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '57d28e1bf8b247f8887df8135d92a40b', 'request_id': 'req-a3bdd303-8396-4a75-ab82-0ded96a26661'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:41.515639 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-a3bdd303-8396-4a75-ab82-0ded96a26661 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '57d28e1bf8b247f8887df8135d92a40b', 'request_id': 'req-a3bdd303-8396-4a75-ab82-0ded96a26661'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:41.525857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-a3bdd303-8396-4a75-ab82-0ded96a26661 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '57d28e1bf8b247f8887df8135d92a40b', 'request_id': 'req-a3bdd303-8396-4a75-ab82-0ded96a26661'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:41.540920 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-a3bdd303-8396-4a75-ab82-0ded96a26661 admin admin] The token cache is being invalidate because project 57d28e1bf8b247f8887df8135d92a40b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:52:41.541262 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-a3bdd303-8396-4a75-ab82-0ded96a26661 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:41.541975 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-a3bdd303-8396-4a75-ab82-0ded96a26661 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:41.543579 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1217/2453] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:52:41 2022] DELETE /identity/v3/projects/57d28e1bf8b247f8887df8135d92a40b => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:52:41.548977 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905746b9-fec1-481f-8f6f-01d73ed28d0e None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:41.549613 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905746b9-fec1-481f-8f6f-01d73ed28d0e None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:41.549613 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905746b9-fec1-481f-8f6f-01d73ed28d0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:41.579335 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-905746b9-fec1-481f-8f6f-01d73ed28d0e None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:52:41.581452 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-905746b9-fec1-481f-8f6f-01d73ed28d0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:41.651518 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:INSERT=2,keystone:DELETE=13,keystone:SELECT=85 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:52:41.683516 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1237/2454] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:52:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:52:41.695969 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef71a646-8029-42fe-8d75-17bfb20470b3 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:41.696670 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef71a646-8029-42fe-8d75-17bfb20470b3 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:41.697456 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef71a646-8029-42fe-8d75-17bfb20470b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:41.725293 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-ef71a646-8029-42fe-8d75-17bfb20470b3 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:52:41.726973 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-ef71a646-8029-42fe-8d75-17bfb20470b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:41.737763 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1218/2455] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:52:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:52:41.742640 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905746b9-fec1-481f-8f6f-01d73ed28d0e None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:41.756876 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905746b9-fec1-481f-8f6f-01d73ed28d0e None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:41.762774 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9bf119-e594-44fe-ab27-7291d9b0a09b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:41.763519 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9bf119-e594-44fe-ab27-7291d9b0a09b admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:41.764736 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9bf119-e594-44fe-ab27-7291d9b0a09b admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:41.765134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9bf119-e594-44fe-ab27-7291d9b0a09b admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:41.766033 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9bf119-e594-44fe-ab27-7291d9b0a09b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:52:41.766641 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9bf119-e594-44fe-ab27-7291d9b0a09b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:41.768484 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9bf119-e594-44fe-ab27-7291d9b0a09b admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:41.783798 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1238/2456] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:52:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:52:41.788351 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef71a646-8029-42fe-8d75-17bfb20470b3 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:41.798685 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef71a646-8029-42fe-8d75-17bfb20470b3 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:41.804983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae773b6-d7ac-4616-a3d2-daa13c0fc830 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:41.805771 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae773b6-d7ac-4616-a3d2-daa13c0fc830 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:41.806094 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae773b6-d7ac-4616-a3d2-daa13c0fc830 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:41.806487 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae773b6-d7ac-4616-a3d2-daa13c0fc830 admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:41.807422 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae773b6-d7ac-4616-a3d2-daa13c0fc830 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:41.809611 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae773b6-d7ac-4616-a3d2-daa13c0fc830 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:41.854832 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1219/2457] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:52:41 2022] POST /identity/v3/projects => generated 367 bytes in 67 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:52:41.859809 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9bf119-e594-44fe-ab27-7291d9b0a09b admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:41.876173 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9bf119-e594-44fe-ab27-7291d9b0a09b admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:41.956371 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65dbcbf9-3c0c-4dcc-9fa4-2a53e676b420 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:41.957416 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65dbcbf9-3c0c-4dcc-9fa4-2a53e676b420 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:41.957915 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65dbcbf9-3c0c-4dcc-9fa4-2a53e676b420 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:41.958446 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65dbcbf9-3c0c-4dcc-9fa4-2a53e676b420 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:41.959984 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65dbcbf9-3c0c-4dcc-9fa4-2a53e676b420 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:41.963762 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65dbcbf9-3c0c-4dcc-9fa4-2a53e676b420 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:42.008404 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1239/2458] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:52:41 2022] POST /identity/v3/users => generated 284 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:52:42.014581 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae773b6-d7ac-4616-a3d2-daa13c0fc830 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:42.030443 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae773b6-d7ac-4616-a3d2-daa13c0fc830 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:42.036827 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356211e2-4a9c-44b2-8f02-315ea415f89e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:42.037736 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356211e2-4a9c-44b2-8f02-315ea415f89e admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:42.038144 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356211e2-4a9c-44b2-8f02-315ea415f89e admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:42.038576 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356211e2-4a9c-44b2-8f02-315ea415f89e admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:42.039578 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356211e2-4a9c-44b2-8f02-315ea415f89e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:42.041682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356211e2-4a9c-44b2-8f02-315ea415f89e admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:42.054244 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1220/2459] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:52:42 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:52:42.058600 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65dbcbf9-3c0c-4dcc-9fa4-2a53e676b420 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:42.075389 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65dbcbf9-3c0c-4dcc-9fa4-2a53e676b420 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:42.085102 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be444bb2-aba8-43c8-bd56-041ededbf90d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:42.086170 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be444bb2-aba8-43c8-bd56-041ededbf90d admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:42.086743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be444bb2-aba8-43c8-bd56-041ededbf90d admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:42.087244 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be444bb2-aba8-43c8-bd56-041ededbf90d admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:42.088443 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be444bb2-aba8-43c8-bd56-041ededbf90d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:42.091600 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be444bb2-aba8-43c8-bd56-041ededbf90d admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:42.111805 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1240/2460] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:52:42 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:52:42.117973 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356211e2-4a9c-44b2-8f02-315ea415f89e admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:42.138462 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356211e2-4a9c-44b2-8f02-315ea415f89e admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:42.149808 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f3a0cb-8901-4753-b865-1b0639f58679 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:42.151022 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f3a0cb-8901-4753-b865-1b0639f58679 admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:42.151512 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f3a0cb-8901-4753-b865-1b0639f58679 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:42.152022 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f3a0cb-8901-4753-b865-1b0639f58679 admin admin] PATH_INFO: `/v3/projects/26cc37255bb34b02941e9fb18c753762/users/63eca654a8bb4c0c89cb6d42ddda063a/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:42.180439 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f3a0cb-8901-4753-b865-1b0639f58679 admin admin] RBAC: Authorizing `identity:create_grant(project_id=26cc37255bb34b02941e9fb18c753762, user_id=63eca654a8bb4c0c89cb6d42ddda063a, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:42.183610 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f3a0cb-8901-4753-b865-1b0639f58679 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:42.200814 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1221/2461] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:52:42 2022] PUT /identity/v3/projects/26cc37255bb34b02941e9fb18c753762/users/63eca654a8bb4c0c89cb6d42ddda063a/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 83 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:52:42.212488 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bab4d2d-0404-45cc-b379-eb426d7b06a9 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:42.212984 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bab4d2d-0404-45cc-b379-eb426d7b06a9 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:42.213473 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bab4d2d-0404-45cc-b379-eb426d7b06a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:42.252012 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-4bab4d2d-0404-45cc-b379-eb426d7b06a9 None None] MFA Rules not processed for user `63eca654a8bb4c0c89cb6d42ddda063a`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:52:42.254227 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4bab4d2d-0404-45cc-b379-eb426d7b06a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:42.350776 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1241/2462] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:52:42 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:52:42.356421 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f3a0cb-8901-4753-b865-1b0639f58679 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:42.362615 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:52:42.450640 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f3a0cb-8901-4753-b865-1b0639f58679 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:42.455866 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54772eb-ccad-445e-9b79-2eab06b3c06b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:42.456650 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54772eb-ccad-445e-9b79-2eab06b3c06b admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:42.457030 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54772eb-ccad-445e-9b79-2eab06b3c06b admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:42.457404 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54772eb-ccad-445e-9b79-2eab06b3c06b admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:42.458143 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d54772eb-ccad-445e-9b79-2eab06b3c06b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:42.460320 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d54772eb-ccad-445e-9b79-2eab06b3c06b admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:42.514248 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1222/2463] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:52:42 2022] POST /identity/v3/projects => generated 367 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:52:42.515496 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bab4d2d-0404-45cc-b379-eb426d7b06a9 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:42.537392 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bab4d2d-0404-45cc-b379-eb426d7b06a9 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:42.605033 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:52:42.638012 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c26227d-77ca-4616-b9db-087ec387a72b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:42.639001 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c26227d-77ca-4616-b9db-087ec387a72b admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:42.639392 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c26227d-77ca-4616-b9db-087ec387a72b admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:42.639763 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c26227d-77ca-4616-b9db-087ec387a72b admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:42.640542 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c26227d-77ca-4616-b9db-087ec387a72b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:42.642714 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c26227d-77ca-4616-b9db-087ec387a72b admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:42.678738 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1242/2464] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:52:42 2022] POST /identity/v3/users => generated 290 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:52:42.681150 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54772eb-ccad-445e-9b79-2eab06b3c06b admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:42.697689 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54772eb-ccad-445e-9b79-2eab06b3c06b admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:42.705322 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c924bebc-696f-4af8-bafa-8705edfa85f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:42.706267 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c924bebc-696f-4af8-bafa-8705edfa85f1 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:42.706858 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c924bebc-696f-4af8-bafa-8705edfa85f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:42.707302 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c924bebc-696f-4af8-bafa-8705edfa85f1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:42.708420 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c924bebc-696f-4af8-bafa-8705edfa85f1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:42.711347 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c924bebc-696f-4af8-bafa-8705edfa85f1 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:42.726690 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1223/2465] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:52:42 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:52:42.733576 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c26227d-77ca-4616-b9db-087ec387a72b admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:42.746925 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c26227d-77ca-4616-b9db-087ec387a72b admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:42.752941 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba26324-b0fb-4f18-aaee-68e414b055b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:42.753781 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba26324-b0fb-4f18-aaee-68e414b055b1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:42.754160 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba26324-b0fb-4f18-aaee-68e414b055b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:42.754594 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba26324-b0fb-4f18-aaee-68e414b055b1 admin admin] PATH_INFO: `/v3/projects/a61b4331030145e4a2b3b6697eb1b2ae/users/4a4b1059f3544f1ab4642c8cf5de4aba/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:42.777781 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba26324-b0fb-4f18-aaee-68e414b055b1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a61b4331030145e4a2b3b6697eb1b2ae, user_id=4a4b1059f3544f1ab4642c8cf5de4aba, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:42.781576 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba26324-b0fb-4f18-aaee-68e414b055b1 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:42.809382 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1243/2466] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:52:42 2022] PUT /identity/v3/projects/a61b4331030145e4a2b3b6697eb1b2ae/users/4a4b1059f3544f1ab4642c8cf5de4aba/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:52:42.810579 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c924bebc-696f-4af8-bafa-8705edfa85f1 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:42.822063 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c924bebc-696f-4af8-bafa-8705edfa85f1 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:42.899209 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9361a456-ec62-469b-b1c0-18cc9c831fa4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:42.900001 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9361a456-ec62-469b-b1c0-18cc9c831fa4 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:42.900281 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9361a456-ec62-469b-b1c0-18cc9c831fa4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:42.900554 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9361a456-ec62-469b-b1c0-18cc9c831fa4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:42.901225 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9361a456-ec62-469b-b1c0-18cc9c831fa4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:42.902931 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9361a456-ec62-469b-b1c0-18cc9c831fa4 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:42.916170 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1224/2467] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:52:42 2022] GET /identity/v3/roles => generated 1420 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:52:42.926994 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba26324-b0fb-4f18-aaee-68e414b055b1 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:42.945835 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba26324-b0fb-4f18-aaee-68e414b055b1 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:42.954875 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-917532fd-361b-4a72-bf68-035ae660afd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:42.956052 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917532fd-361b-4a72-bf68-035ae660afd8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:42.956553 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917532fd-361b-4a72-bf68-035ae660afd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:42.957036 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917532fd-361b-4a72-bf68-035ae660afd8 admin admin] PATH_INFO: `/v3/projects/a61b4331030145e4a2b3b6697eb1b2ae/users/4a4b1059f3544f1ab4642c8cf5de4aba/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:42.960530 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-917532fd-361b-4a72-bf68-035ae660afd8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a61b4331030145e4a2b3b6697eb1b2ae, user_id=4a4b1059f3544f1ab4642c8cf5de4aba, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:42.965451 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-917532fd-361b-4a72-bf68-035ae660afd8 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:42.984700 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1244/2468] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:52:42 2022] PUT /identity/v3/projects/a61b4331030145e4a2b3b6697eb1b2ae/users/4a4b1059f3544f1ab4642c8cf5de4aba/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:52:42.996662 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fd37f2-1b65-4509-ac2d-5c85c8137667 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:42.997279 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fd37f2-1b65-4509-ac2d-5c85c8137667 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:42.997655 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fd37f2-1b65-4509-ac2d-5c85c8137667 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:43.044532 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-e4fd37f2-1b65-4509-ac2d-5c85c8137667 None None] MFA Rules not processed for user `4a4b1059f3544f1ab4642c8cf5de4aba`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:52:43.045970 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-e4fd37f2-1b65-4509-ac2d-5c85c8137667 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:43.150826 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:52:43.161404 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1225/2469] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:52:42 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:52:43.180531 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-917532fd-361b-4a72-bf68-035ae660afd8 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:43.183179 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-917532fd-361b-4a72-bf68-035ae660afd8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:43.278191 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-917532fd-361b-4a72-bf68-035ae660afd8 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:43.285023 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec53e61-65a5-4ff3-b0a0-059e6303b5f2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:43.286003 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec53e61-65a5-4ff3-b0a0-059e6303b5f2 service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:43.286496 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec53e61-65a5-4ff3-b0a0-059e6303b5f2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:43.286947 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec53e61-65a5-4ff3-b0a0-059e6303b5f2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:43.302699 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ec53e61-65a5-4ff3-b0a0-059e6303b5f2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:43.306009 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ec53e61-65a5-4ff3-b0a0-059e6303b5f2 service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:43.321017 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1245/2470] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:52:43 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:52:43.395436 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4fd37f2-1b65-4509-ac2d-5c85c8137667 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:43.397337 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-e4fd37f2-1b65-4509-ac2d-5c85c8137667 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:43.412545 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4fd37f2-1b65-4509-ac2d-5c85c8137667 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:43.417467 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acac416e-b03e-490f-b01e-1a8932b9f476 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:43.418421 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acac416e-b03e-490f-b01e-1a8932b9f476 service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:43.418905 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acac416e-b03e-490f-b01e-1a8932b9f476 service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:43.419636 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acac416e-b03e-490f-b01e-1a8932b9f476 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:43.486645 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec53e61-65a5-4ff3-b0a0-059e6303b5f2 service cinder] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:43.489627 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-6ec53e61-65a5-4ff3-b0a0-059e6303b5f2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:43.542932 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acac416e-b03e-490f-b01e-1a8932b9f476 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:43.547306 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acac416e-b03e-490f-b01e-1a8932b9f476 service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:43.588616 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1226/2471] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:52:43 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 194 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:52:43.605614 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec53e61-65a5-4ff3-b0a0-059e6303b5f2 service cinder] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:43.612273 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b97fb7a-713a-43db-8aa1-2b9586df1a93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:43.612937 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b97fb7a-713a-43db-8aa1-2b9586df1a93 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:43.613215 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b97fb7a-713a-43db-8aa1-2b9586df1a93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:43.613492 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b97fb7a-713a-43db-8aa1-2b9586df1a93 admin admin] PATH_INFO: `/v3/users/1ace0b58f73c4279adf024cc677330bd` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:43.614747 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b97fb7a-713a-43db-8aa1-2b9586df1a93 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1ace0b58f73c4279adf024cc677330bd)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:43.616712 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b97fb7a-713a-43db-8aa1-2b9586df1a93 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:43.679518 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=90,keystone:INSERT=5,keystone:DELETE=5 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:52:43.706028 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7b97fb7a-713a-43db-8aa1-2b9586df1a93 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1ace0b58f73c4279adf024cc677330bd'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:43.720300 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7b97fb7a-713a-43db-8aa1-2b9586df1a93 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1ace0b58f73c4279adf024cc677330bd'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:43.721653 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7b97fb7a-713a-43db-8aa1-2b9586df1a93 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1ace0b58f73c4279adf024cc677330bd'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:43.722798 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7b97fb7a-713a-43db-8aa1-2b9586df1a93 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1ace0b58f73c4279adf024cc677330bd'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:43.737902 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7b97fb7a-713a-43db-8aa1-2b9586df1a93 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1ace0b58f73c4279adf024cc677330bd'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:43.772006 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1246/2472] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:52:43 2022] DELETE /identity/v3/users/1ace0b58f73c4279adf024cc677330bd => generated 0 bytes in 287 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:52:43.779881 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fabb2fea-ca02-4bbf-a035-2e64438707ce None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:43.780774 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fabb2fea-ca02-4bbf-a035-2e64438707ce None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:43.781002 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fabb2fea-ca02-4bbf-a035-2e64438707ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:43.818576 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-fabb2fea-ca02-4bbf-a035-2e64438707ce None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:52:43.819898 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-fabb2fea-ca02-4bbf-a035-2e64438707ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:43.898544 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1227/2473] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:52:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:52:43.909541 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b97fb7a-713a-43db-8aa1-2b9586df1a93 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:43.912318 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7b97fb7a-713a-43db-8aa1-2b9586df1a93 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:44.037460 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b97fb7a-713a-43db-8aa1-2b9586df1a93 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:44.047355 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ff69b8-f006-4c93-ad78-863ea9548ace service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:44.048547 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ff69b8-f006-4c93-ad78-863ea9548ace service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:44.049239 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ff69b8-f006-4c93-ad78-863ea9548ace service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:44.050127 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ff69b8-f006-4c93-ad78-863ea9548ace service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:44.082897 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64ff69b8-f006-4c93-ad78-863ea9548ace service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:44.086754 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64ff69b8-f006-4c93-ad78-863ea9548ace service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:44.110807 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1247/2474] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:52:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 202 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:52:44.498643 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fabb2fea-ca02-4bbf-a035-2e64438707ce None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:44.500166 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-fabb2fea-ca02-4bbf-a035-2e64438707ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:44.515214 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fabb2fea-ca02-4bbf-a035-2e64438707ce None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:44.520577 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a96946-2b77-46b7-86d4-cea9d5573644 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:44.521148 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a96946-2b77-46b7-86d4-cea9d5573644 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:44.521429 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a96946-2b77-46b7-86d4-cea9d5573644 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:44.521684 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a96946-2b77-46b7-86d4-cea9d5573644 admin admin] PATH_INFO: `/v3/projects/cc0b8c4858ad4ccbb73c54cd7012d924` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:44.523136 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16a96946-2b77-46b7-86d4-cea9d5573644 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cc0b8c4858ad4ccbb73c54cd7012d924)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:44.525097 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16a96946-2b77-46b7-86d4-cea9d5573644 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:44.586544 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-16a96946-2b77-46b7-86d4-cea9d5573644 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cc0b8c4858ad4ccbb73c54cd7012d924', 'request_id': 'req-16a96946-2b77-46b7-86d4-cea9d5573644'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:44.596345 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-16a96946-2b77-46b7-86d4-cea9d5573644 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cc0b8c4858ad4ccbb73c54cd7012d924', 'request_id': 'req-16a96946-2b77-46b7-86d4-cea9d5573644'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:44.608985 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-16a96946-2b77-46b7-86d4-cea9d5573644 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cc0b8c4858ad4ccbb73c54cd7012d924', 'request_id': 'req-16a96946-2b77-46b7-86d4-cea9d5573644'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:44.627290 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-16a96946-2b77-46b7-86d4-cea9d5573644 admin admin] The token cache is being invalidate because project cc0b8c4858ad4ccbb73c54cd7012d924 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:52:44.627729 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-16a96946-2b77-46b7-86d4-cea9d5573644 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:44.628453 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-16a96946-2b77-46b7-86d4-cea9d5573644 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:44.630219 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1228/2475] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:52:44 2022] DELETE /identity/v3/projects/cc0b8c4858ad4ccbb73c54cd7012d924 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:52:44.637517 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfbe71f-1434-4630-9783-ed8fe6022948 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:44.638184 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfbe71f-1434-4630-9783-ed8fe6022948 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:44.639501 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bfbe71f-1434-4630-9783-ed8fe6022948 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:44.668052 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-2bfbe71f-1434-4630-9783-ed8fe6022948 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:52:44.669553 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-2bfbe71f-1434-4630-9783-ed8fe6022948 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:44.770840 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1248/2476] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:52:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:52:44.780430 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc2bd0a-ac26-4651-8a47-0f93328167e4 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:44.781388 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc2bd0a-ac26-4651-8a47-0f93328167e4 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:44.781583 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc2bd0a-ac26-4651-8a47-0f93328167e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:44.805516 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-7cc2bd0a-ac26-4651-8a47-0f93328167e4 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:52:44.806839 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-7cc2bd0a-ac26-4651-8a47-0f93328167e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:44.815997 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1229/2477] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:52:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:52:44.821887 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bfbe71f-1434-4630-9783-ed8fe6022948 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:44.834081 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bfbe71f-1434-4630-9783-ed8fe6022948 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:44.839892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b6f348-9824-4b2d-a754-bc2376619c8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:44.840442 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b6f348-9824-4b2d-a754-bc2376619c8d admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:44.840722 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b6f348-9824-4b2d-a754-bc2376619c8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:44.841004 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b6f348-9824-4b2d-a754-bc2376619c8d admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:44.841694 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b6f348-9824-4b2d-a754-bc2376619c8d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:52:44.842031 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b6f348-9824-4b2d-a754-bc2376619c8d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:44.843727 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b6f348-9824-4b2d-a754-bc2376619c8d admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:44.865478 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1249/2478] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:52:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:52:44.867401 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc2bd0a-ac26-4651-8a47-0f93328167e4 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:44.886299 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc2bd0a-ac26-4651-8a47-0f93328167e4 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:44.892648 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad987e2b-243f-4f07-946e-e51a4d1a7fce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:44.893436 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad987e2b-243f-4f07-946e-e51a4d1a7fce admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:44.893829 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad987e2b-243f-4f07-946e-e51a4d1a7fce admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:44.894199 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad987e2b-243f-4f07-946e-e51a4d1a7fce admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:44.894992 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad987e2b-243f-4f07-946e-e51a4d1a7fce admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:44.897459 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad987e2b-243f-4f07-946e-e51a4d1a7fce admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:44.914302 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:52:44.942610 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1230/2479] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:52:44 2022] POST /identity/v3/projects => generated 373 bytes in 74 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:52:44.946911 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b6f348-9824-4b2d-a754-bc2376619c8d admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:44.963904 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b6f348-9824-4b2d-a754-bc2376619c8d admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:45.027168 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:52:45.055222 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c878e187-da59-4142-92d7-79be9608da6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:45.055912 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c878e187-da59-4142-92d7-79be9608da6b admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:45.056336 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c878e187-da59-4142-92d7-79be9608da6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:45.056499 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c878e187-da59-4142-92d7-79be9608da6b admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:45.057273 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c878e187-da59-4142-92d7-79be9608da6b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:45.059738 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c878e187-da59-4142-92d7-79be9608da6b admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:45.093623 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1250/2480] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:52:44 2022] POST /identity/v3/users => generated 287 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:52:45.094982 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad987e2b-243f-4f07-946e-e51a4d1a7fce admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:45.110719 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad987e2b-243f-4f07-946e-e51a4d1a7fce admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:45.116845 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89dc5bb9-7d5f-40a3-bf42-e8d0a33bf10f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:45.117985 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89dc5bb9-7d5f-40a3-bf42-e8d0a33bf10f admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:45.118580 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89dc5bb9-7d5f-40a3-bf42-e8d0a33bf10f admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:45.118853 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89dc5bb9-7d5f-40a3-bf42-e8d0a33bf10f admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:45.119597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89dc5bb9-7d5f-40a3-bf42-e8d0a33bf10f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:45.121510 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89dc5bb9-7d5f-40a3-bf42-e8d0a33bf10f admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:45.147611 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1231/2481] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:52:45 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:52:45.151598 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c878e187-da59-4142-92d7-79be9608da6b admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:45.162840 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c878e187-da59-4142-92d7-79be9608da6b admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:45.174662 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc165555-0923-4e23-9fdb-16b9c07b6bb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:45.175731 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc165555-0923-4e23-9fdb-16b9c07b6bb8 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:45.176240 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc165555-0923-4e23-9fdb-16b9c07b6bb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:45.176758 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc165555-0923-4e23-9fdb-16b9c07b6bb8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:45.177948 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc165555-0923-4e23-9fdb-16b9c07b6bb8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:45.181304 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc165555-0923-4e23-9fdb-16b9c07b6bb8 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:45.215383 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1251/2482] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:52:45 2022] GET /identity/v3/roles => generated 1420 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:52:45.216989 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89dc5bb9-7d5f-40a3-bf42-e8d0a33bf10f admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:45.233620 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89dc5bb9-7d5f-40a3-bf42-e8d0a33bf10f admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:45.241938 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81331135-2d8b-45aa-a7fa-f3e04bc43361 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:45.243510 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81331135-2d8b-45aa-a7fa-f3e04bc43361 admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:45.243510 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81331135-2d8b-45aa-a7fa-f3e04bc43361 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:45.243510 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81331135-2d8b-45aa-a7fa-f3e04bc43361 admin admin] PATH_INFO: `/v3/projects/5cbdd43cc2174746804966f7a7eabd9d/users/23aa9c4ec665498ebd86db39135f696f/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:45.270079 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81331135-2d8b-45aa-a7fa-f3e04bc43361 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5cbdd43cc2174746804966f7a7eabd9d, user_id=23aa9c4ec665498ebd86db39135f696f, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:45.272812 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81331135-2d8b-45aa-a7fa-f3e04bc43361 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:45.297053 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1232/2483] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:52:45 2022] PUT /identity/v3/projects/5cbdd43cc2174746804966f7a7eabd9d/users/23aa9c4ec665498ebd86db39135f696f/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:52:45.309342 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4eae8e-bc83-4746-958c-b0d3f6c19ef2 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:45.309876 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4eae8e-bc83-4746-958c-b0d3f6c19ef2 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:45.310062 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4eae8e-bc83-4746-958c-b0d3f6c19ef2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:45.346749 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-3e4eae8e-bc83-4746-958c-b0d3f6c19ef2 None None] MFA Rules not processed for user `23aa9c4ec665498ebd86db39135f696f`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:52:45.347918 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-3e4eae8e-bc83-4746-958c-b0d3f6c19ef2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:45.506319 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1252/2484] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:52:45 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:52:45.508114 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81331135-2d8b-45aa-a7fa-f3e04bc43361 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:45.643207 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81331135-2d8b-45aa-a7fa-f3e04bc43361 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:45.653082 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6910e3d-1dad-4649-b0a3-e86d364f1bef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:45.654175 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6910e3d-1dad-4649-b0a3-e86d364f1bef admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:45.655382 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6910e3d-1dad-4649-b0a3-e86d364f1bef admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:45.655879 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6910e3d-1dad-4649-b0a3-e86d364f1bef admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:45.657070 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6910e3d-1dad-4649-b0a3-e86d364f1bef admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:45.661265 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6910e3d-1dad-4649-b0a3-e86d364f1bef admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:45.716645 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1233/2485] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:52:45 2022] POST /identity/v3/projects => generated 369 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:52:45.722829 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e4eae8e-bc83-4746-958c-b0d3f6c19ef2 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:45.739459 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e4eae8e-bc83-4746-958c-b0d3f6c19ef2 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:45.957192 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df67ba66-4b77-4c36-973e-12df3cb10a39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:45.958709 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df67ba66-4b77-4c36-973e-12df3cb10a39 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:45.959185 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df67ba66-4b77-4c36-973e-12df3cb10a39 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:45.959539 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df67ba66-4b77-4c36-973e-12df3cb10a39 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:45.960364 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df67ba66-4b77-4c36-973e-12df3cb10a39 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:45.963693 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df67ba66-4b77-4c36-973e-12df3cb10a39 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:46.004156 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1253/2486] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:52:45 2022] POST /identity/v3/users => generated 291 bytes in 282 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:52:46.009729 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6910e3d-1dad-4649-b0a3-e86d364f1bef admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:46.028527 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6910e3d-1dad-4649-b0a3-e86d364f1bef admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:46.060008 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d1d23a-99c2-4b75-b93a-65e0ec4b2c2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:46.060864 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d1d23a-99c2-4b75-b93a-65e0ec4b2c2f admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:46.061260 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d1d23a-99c2-4b75-b93a-65e0ec4b2c2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:46.061647 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d1d23a-99c2-4b75-b93a-65e0ec4b2c2f admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:46.062623 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d1d23a-99c2-4b75-b93a-65e0ec4b2c2f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:46.065025 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d1d23a-99c2-4b75-b93a-65e0ec4b2c2f admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:46.091247 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1234/2487] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:52:46 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:52:46.096968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df67ba66-4b77-4c36-973e-12df3cb10a39 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:46.110538 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df67ba66-4b77-4c36-973e-12df3cb10a39 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:46.116025 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae4d6e2-cbe3-4dd4-a7aa-0626de21613a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:46.116732 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae4d6e2-cbe3-4dd4-a7aa-0626de21613a admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:46.117017 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae4d6e2-cbe3-4dd4-a7aa-0626de21613a admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:46.117297 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae4d6e2-cbe3-4dd4-a7aa-0626de21613a admin admin] PATH_INFO: `/v3/projects/a1885f63b4014df7bc4d675070603132/users/7b46347b8afc4e149f9a37642ef85c6d/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:46.135906 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae4d6e2-cbe3-4dd4-a7aa-0626de21613a admin admin] RBAC: Authorizing `identity:create_grant(project_id=a1885f63b4014df7bc4d675070603132, user_id=7b46347b8afc4e149f9a37642ef85c6d, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:46.139692 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae4d6e2-cbe3-4dd4-a7aa-0626de21613a admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:46.160087 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1254/2488] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:52:46 2022] PUT /identity/v3/projects/a1885f63b4014df7bc4d675070603132/users/7b46347b8afc4e149f9a37642ef85c6d/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:52:46.164452 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d1d23a-99c2-4b75-b93a-65e0ec4b2c2f admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:46.175514 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d1d23a-99c2-4b75-b93a-65e0ec4b2c2f admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:46.254244 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c35593b-50e5-4c94-bdd8-0307163c725a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:46.254944 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c35593b-50e5-4c94-bdd8-0307163c725a admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:46.255222 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c35593b-50e5-4c94-bdd8-0307163c725a admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:46.255495 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c35593b-50e5-4c94-bdd8-0307163c725a admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:46.256157 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c35593b-50e5-4c94-bdd8-0307163c725a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:46.257850 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c35593b-50e5-4c94-bdd8-0307163c725a admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:46.270766 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1235/2489] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:52:46 2022] GET /identity/v3/roles => generated 1420 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:52:46.277231 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae4d6e2-cbe3-4dd4-a7aa-0626de21613a admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:46.290410 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae4d6e2-cbe3-4dd4-a7aa-0626de21613a admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:46.300089 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0c7b66-375c-4db4-bd9a-eaed2eda88b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:46.301430 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0c7b66-375c-4db4-bd9a-eaed2eda88b8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:46.302094 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0c7b66-375c-4db4-bd9a-eaed2eda88b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:46.302774 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0c7b66-375c-4db4-bd9a-eaed2eda88b8 admin admin] PATH_INFO: `/v3/projects/a1885f63b4014df7bc4d675070603132/users/7b46347b8afc4e149f9a37642ef85c6d/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:46.307719 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a0c7b66-375c-4db4-bd9a-eaed2eda88b8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a1885f63b4014df7bc4d675070603132, user_id=7b46347b8afc4e149f9a37642ef85c6d, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:46.313563 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a0c7b66-375c-4db4-bd9a-eaed2eda88b8 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:46.338130 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1255/2490] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:52:46 2022] PUT /identity/v3/projects/a1885f63b4014df7bc4d675070603132/users/7b46347b8afc4e149f9a37642ef85c6d/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:52:46.355512 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0776d913-71bc-4e21-a7fc-ab73e720d48f None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:46.356033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0776d913-71bc-4e21-a7fc-ab73e720d48f None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:46.356488 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0776d913-71bc-4e21-a7fc-ab73e720d48f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:46.405142 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-0776d913-71bc-4e21-a7fc-ab73e720d48f None None] MFA Rules not processed for user `7b46347b8afc4e149f9a37642ef85c6d`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:52:46.407380 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-0776d913-71bc-4e21-a7fc-ab73e720d48f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:46.433060 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:52:46.522613 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1236/2491] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:52:46 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:52:46.539239 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0c7b66-375c-4db4-bd9a-eaed2eda88b8 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:46.542368 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4a0c7b66-375c-4db4-bd9a-eaed2eda88b8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:46.554103 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:52:46.638848 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0c7b66-375c-4db4-bd9a-eaed2eda88b8 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:46.642837 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f44bed-b6d9-4812-ba7d-53bb0f32553b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:46.643558 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f44bed-b6d9-4812-ba7d-53bb0f32553b service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:46.643925 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f44bed-b6d9-4812-ba7d-53bb0f32553b service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:46.644257 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f44bed-b6d9-4812-ba7d-53bb0f32553b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:46.655286 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83f44bed-b6d9-4812-ba7d-53bb0f32553b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:46.656855 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83f44bed-b6d9-4812-ba7d-53bb0f32553b service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:46.667712 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1256/2492] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:52:46 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:52:47.356222 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0776d913-71bc-4e21-a7fc-ab73e720d48f None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:47.509863 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0776d913-71bc-4e21-a7fc-ab73e720d48f None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:47.526582 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd579cd8-c7fb-4b0e-9050-132281f80da8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:47.528038 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd579cd8-c7fb-4b0e-9050-132281f80da8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:47.528038 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd579cd8-c7fb-4b0e-9050-132281f80da8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:47.528306 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd579cd8-c7fb-4b0e-9050-132281f80da8 admin admin] PATH_INFO: `/v3/users/23aa9c4ec665498ebd86db39135f696f` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:47.533211 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd579cd8-c7fb-4b0e-9050-132281f80da8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=23aa9c4ec665498ebd86db39135f696f)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:47.539160 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd579cd8-c7fb-4b0e-9050-132281f80da8 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:47.647213 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-fd579cd8-c7fb-4b0e-9050-132281f80da8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '23aa9c4ec665498ebd86db39135f696f'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:47.662899 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-fd579cd8-c7fb-4b0e-9050-132281f80da8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '23aa9c4ec665498ebd86db39135f696f'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:47.664302 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-fd579cd8-c7fb-4b0e-9050-132281f80da8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '23aa9c4ec665498ebd86db39135f696f'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:47.666374 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-fd579cd8-c7fb-4b0e-9050-132281f80da8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '23aa9c4ec665498ebd86db39135f696f'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:47.680862 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-fd579cd8-c7fb-4b0e-9050-132281f80da8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '23aa9c4ec665498ebd86db39135f696f'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:47.702785 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1237/2493] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:52:47 2022] DELETE /identity/v3/users/23aa9c4ec665498ebd86db39135f696f => generated 0 bytes in 347 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:52:47.707258 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57ccf46-fc7b-4fc8-bdd5-029aa256afd9 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:47.707814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57ccf46-fc7b-4fc8-bdd5-029aa256afd9 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:47.707814 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57ccf46-fc7b-4fc8-bdd5-029aa256afd9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:47.733449 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-f57ccf46-fc7b-4fc8-bdd5-029aa256afd9 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:52:47.736870 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-f57ccf46-fc7b-4fc8-bdd5-029aa256afd9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:47.850149 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1257/2494] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:52:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:52:47.851836 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd579cd8-c7fb-4b0e-9050-132281f80da8 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:47.853563 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-fd579cd8-c7fb-4b0e-9050-132281f80da8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:47.862574 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f57ccf46-fc7b-4fc8-bdd5-029aa256afd9 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:47.864767 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-f57ccf46-fc7b-4fc8-bdd5-029aa256afd9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:47.885160 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd579cd8-c7fb-4b0e-9050-132281f80da8 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:47.895844 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f821b7-d43a-43fd-acca-d5d5c00d6b8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:47.896920 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f821b7-d43a-43fd-acca-d5d5c00d6b8a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:47.897343 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f821b7-d43a-43fd-acca-d5d5c00d6b8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:47.897811 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f821b7-d43a-43fd-acca-d5d5c00d6b8a admin admin] PATH_INFO: `/v3/users/63eca654a8bb4c0c89cb6d42ddda063a` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:47.900005 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f821b7-d43a-43fd-acca-d5d5c00d6b8a admin admin] RBAC: Authorizing `identity:delete_user(user_id=63eca654a8bb4c0c89cb6d42ddda063a)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:47.904398 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f821b7-d43a-43fd-acca-d5d5c00d6b8a admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:47.949581 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:52:47.982576 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-58f821b7-d43a-43fd-acca-d5d5c00d6b8a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '63eca654a8bb4c0c89cb6d42ddda063a'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:47.982880 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f57ccf46-fc7b-4fc8-bdd5-029aa256afd9 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:47.989980 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67eeb5b0-57b3-4d80-8e14-f799cd1e7748 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:47.990557 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67eeb5b0-57b3-4d80-8e14-f799cd1e7748 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:47.990854 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67eeb5b0-57b3-4d80-8e14-f799cd1e7748 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:47.991105 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67eeb5b0-57b3-4d80-8e14-f799cd1e7748 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:47.995964 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-58f821b7-d43a-43fd-acca-d5d5c00d6b8a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '63eca654a8bb4c0c89cb6d42ddda063a'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:47.999031 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-58f821b7-d43a-43fd-acca-d5d5c00d6b8a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '63eca654a8bb4c0c89cb6d42ddda063a'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:48.000935 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-58f821b7-d43a-43fd-acca-d5d5c00d6b8a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '63eca654a8bb4c0c89cb6d42ddda063a'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:48.004872 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67eeb5b0-57b3-4d80-8e14-f799cd1e7748 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:48.007637 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67eeb5b0-57b3-4d80-8e14-f799cd1e7748 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:48.013029 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-58f821b7-d43a-43fd-acca-d5d5c00d6b8a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '63eca654a8bb4c0c89cb6d42ddda063a'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:48.024418 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1258/2495] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:52:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:52:48.057568 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1238/2496] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:52:47 2022] DELETE /identity/v3/users/63eca654a8bb4c0c89cb6d42ddda063a => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:52:48.065986 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc69bff-bedb-4c99-a2a3-d37bd0fb7741 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:48.067056 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc69bff-bedb-4c99-a2a3-d37bd0fb7741 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:48.067926 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc69bff-bedb-4c99-a2a3-d37bd0fb7741 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:48.095509 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-0bc69bff-bedb-4c99-a2a3-d37bd0fb7741 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:52:48.097243 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-0bc69bff-bedb-4c99-a2a3-d37bd0fb7741 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:48.158982 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:52:48.192054 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1259/2497] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:52:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:52:48.207273 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f821b7-d43a-43fd-acca-d5d5c00d6b8a admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:48.208895 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-58f821b7-d43a-43fd-acca-d5d5c00d6b8a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:48.288496 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f821b7-d43a-43fd-acca-d5d5c00d6b8a admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:48.295440 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c51734-c16b-40b5-aa06-441140c2dc32 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:48.296228 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c51734-c16b-40b5-aa06-441140c2dc32 service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:48.297042 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c51734-c16b-40b5-aa06-441140c2dc32 service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:48.297443 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c51734-c16b-40b5-aa06-441140c2dc32 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:48.312540 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c51734-c16b-40b5-aa06-441140c2dc32 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:48.315140 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c51734-c16b-40b5-aa06-441140c2dc32 service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:48.330056 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1239/2498] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:52:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:52:48.448581 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc69bff-bedb-4c99-a2a3-d37bd0fb7741 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:48.450834 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-0bc69bff-bedb-4c99-a2a3-d37bd0fb7741 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:48.474982 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc69bff-bedb-4c99-a2a3-d37bd0fb7741 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:48.483645 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e003cce8-cdd2-4764-bf0b-e93b561463aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:48.486007 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e003cce8-cdd2-4764-bf0b-e93b561463aa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:48.486770 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e003cce8-cdd2-4764-bf0b-e93b561463aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:48.487281 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e003cce8-cdd2-4764-bf0b-e93b561463aa admin admin] PATH_INFO: `/v3/projects/5cbdd43cc2174746804966f7a7eabd9d` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:48.490077 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e003cce8-cdd2-4764-bf0b-e93b561463aa admin admin] RBAC: Authorizing `identity:delete_project(project_id=5cbdd43cc2174746804966f7a7eabd9d)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:48.493504 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e003cce8-cdd2-4764-bf0b-e93b561463aa admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:48.571966 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e003cce8-cdd2-4764-bf0b-e93b561463aa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5cbdd43cc2174746804966f7a7eabd9d', 'request_id': 'req-e003cce8-cdd2-4764-bf0b-e93b561463aa'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:48.588926 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e003cce8-cdd2-4764-bf0b-e93b561463aa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5cbdd43cc2174746804966f7a7eabd9d', 'request_id': 'req-e003cce8-cdd2-4764-bf0b-e93b561463aa'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:48.603104 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e003cce8-cdd2-4764-bf0b-e93b561463aa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5cbdd43cc2174746804966f7a7eabd9d', 'request_id': 'req-e003cce8-cdd2-4764-bf0b-e93b561463aa'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:48.621923 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e003cce8-cdd2-4764-bf0b-e93b561463aa admin admin] The token cache is being invalidate because project 5cbdd43cc2174746804966f7a7eabd9d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:52:48.622216 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e003cce8-cdd2-4764-bf0b-e93b561463aa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:48.624176 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e003cce8-cdd2-4764-bf0b-e93b561463aa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:48.625901 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1260/2499] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:52:48 2022] DELETE /identity/v3/projects/5cbdd43cc2174746804966f7a7eabd9d => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:52:48.632030 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c51734-c16b-40b5-aa06-441140c2dc32 service neutron] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:48.644873 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c51734-c16b-40b5-aa06-441140c2dc32 service neutron] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:48.747416 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275c050d-e973-417d-a6b8-05675b8eebe5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:48.749022 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275c050d-e973-417d-a6b8-05675b8eebe5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:48.749349 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275c050d-e973-417d-a6b8-05675b8eebe5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:48.750386 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275c050d-e973-417d-a6b8-05675b8eebe5 admin admin] PATH_INFO: `/v3/users/7b46347b8afc4e149f9a37642ef85c6d` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:48.751280 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275c050d-e973-417d-a6b8-05675b8eebe5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7b46347b8afc4e149f9a37642ef85c6d)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:48.754228 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275c050d-e973-417d-a6b8-05675b8eebe5 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:48.859735 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-275c050d-e973-417d-a6b8-05675b8eebe5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7b46347b8afc4e149f9a37642ef85c6d'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:48.862606 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:DELETE=13,keystone:SELECT=86,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:52:48.877145 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-275c050d-e973-417d-a6b8-05675b8eebe5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7b46347b8afc4e149f9a37642ef85c6d'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:48.879199 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-275c050d-e973-417d-a6b8-05675b8eebe5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7b46347b8afc4e149f9a37642ef85c6d'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:48.882657 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-275c050d-e973-417d-a6b8-05675b8eebe5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7b46347b8afc4e149f9a37642ef85c6d'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:48.898366 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-275c050d-e973-417d-a6b8-05675b8eebe5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7b46347b8afc4e149f9a37642ef85c6d'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:48.944748 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1240/2500] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:52:48 2022] DELETE /identity/v3/users/7b46347b8afc4e149f9a37642ef85c6d => generated 0 bytes in 314 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:52:49.089506 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e003cce8-cdd2-4764-bf0b-e93b561463aa admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:49.091554 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e003cce8-cdd2-4764-bf0b-e93b561463aa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:49.112134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e003cce8-cdd2-4764-bf0b-e93b561463aa admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:49.192056 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60fabb48-cc2c-4315-8435-39602f59a604 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:49.192938 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60fabb48-cc2c-4315-8435-39602f59a604 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:49.193415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60fabb48-cc2c-4315-8435-39602f59a604 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:49.194429 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60fabb48-cc2c-4315-8435-39602f59a604 admin admin] PATH_INFO: `/v3/projects/26cc37255bb34b02941e9fb18c753762` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:49.195907 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60fabb48-cc2c-4315-8435-39602f59a604 admin admin] RBAC: Authorizing `identity:delete_project(project_id=26cc37255bb34b02941e9fb18c753762)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:49.198119 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60fabb48-cc2c-4315-8435-39602f59a604 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:49.261191 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-60fabb48-cc2c-4315-8435-39602f59a604 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '26cc37255bb34b02941e9fb18c753762', 'request_id': 'req-60fabb48-cc2c-4315-8435-39602f59a604'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:49.268384 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-60fabb48-cc2c-4315-8435-39602f59a604 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '26cc37255bb34b02941e9fb18c753762', 'request_id': 'req-60fabb48-cc2c-4315-8435-39602f59a604'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:49.277015 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-60fabb48-cc2c-4315-8435-39602f59a604 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '26cc37255bb34b02941e9fb18c753762', 'request_id': 'req-60fabb48-cc2c-4315-8435-39602f59a604'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:49.289674 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-60fabb48-cc2c-4315-8435-39602f59a604 admin admin] The token cache is being invalidate because project 26cc37255bb34b02941e9fb18c753762 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:52:49.290001 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-60fabb48-cc2c-4315-8435-39602f59a604 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:49.290682 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-60fabb48-cc2c-4315-8435-39602f59a604 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:49.292413 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1261/2501] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:52:49 2022] DELETE /identity/v3/projects/26cc37255bb34b02941e9fb18c753762 => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:52:49.299915 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275c050d-e973-417d-a6b8-05675b8eebe5 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:49.301581 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-275c050d-e973-417d-a6b8-05675b8eebe5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:49.323774 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275c050d-e973-417d-a6b8-05675b8eebe5 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:49.440847 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08d9a1b3-5de7-4c55-8c31-31651f127924 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:49.441492 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d9a1b3-5de7-4c55-8c31-31651f127924 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:49.442254 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d9a1b3-5de7-4c55-8c31-31651f127924 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:49.442470 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08d9a1b3-5de7-4c55-8c31-31651f127924 admin admin] PATH_INFO: `/v3/users/4a4b1059f3544f1ab4642c8cf5de4aba` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:49.444987 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08d9a1b3-5de7-4c55-8c31-31651f127924 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4a4b1059f3544f1ab4642c8cf5de4aba)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:49.448022 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08d9a1b3-5de7-4c55-8c31-31651f127924 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:49.522812 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-08d9a1b3-5de7-4c55-8c31-31651f127924 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4a4b1059f3544f1ab4642c8cf5de4aba'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:49.539853 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-08d9a1b3-5de7-4c55-8c31-31651f127924 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4a4b1059f3544f1ab4642c8cf5de4aba'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:49.540757 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-08d9a1b3-5de7-4c55-8c31-31651f127924 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4a4b1059f3544f1ab4642c8cf5de4aba'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:49.541735 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-08d9a1b3-5de7-4c55-8c31-31651f127924 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4a4b1059f3544f1ab4642c8cf5de4aba'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:49.563607 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-08d9a1b3-5de7-4c55-8c31-31651f127924 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4a4b1059f3544f1ab4642c8cf5de4aba'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:49.563865 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60fabb48-cc2c-4315-8435-39602f59a604 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:49.566909 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-60fabb48-cc2c-4315-8435-39602f59a604 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:49.593873 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:52:49.600049 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1241/2502] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:52:49 2022] DELETE /identity/v3/users/4a4b1059f3544f1ab4642c8cf5de4aba => generated 0 bytes in 300 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:52:49.603006 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60fabb48-cc2c-4315-8435-39602f59a604 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:49.714231 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-617bd881-13d3-4b51-b587-3240ce5b1064 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:49.715212 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617bd881-13d3-4b51-b587-3240ce5b1064 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:49.715841 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617bd881-13d3-4b51-b587-3240ce5b1064 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:49.716162 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617bd881-13d3-4b51-b587-3240ce5b1064 admin admin] PATH_INFO: `/v3/projects/a1885f63b4014df7bc4d675070603132` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:49.718528 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-617bd881-13d3-4b51-b587-3240ce5b1064 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a1885f63b4014df7bc4d675070603132)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:49.720933 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-617bd881-13d3-4b51-b587-3240ce5b1064 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:49.786872 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-617bd881-13d3-4b51-b587-3240ce5b1064 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a1885f63b4014df7bc4d675070603132', 'request_id': 'req-617bd881-13d3-4b51-b587-3240ce5b1064'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:49.796097 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-617bd881-13d3-4b51-b587-3240ce5b1064 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a1885f63b4014df7bc4d675070603132', 'request_id': 'req-617bd881-13d3-4b51-b587-3240ce5b1064'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:49.809121 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-617bd881-13d3-4b51-b587-3240ce5b1064 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a1885f63b4014df7bc4d675070603132', 'request_id': 'req-617bd881-13d3-4b51-b587-3240ce5b1064'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:49.838777 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-617bd881-13d3-4b51-b587-3240ce5b1064 admin admin] The token cache is being invalidate because project a1885f63b4014df7bc4d675070603132 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:52:49.839357 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-617bd881-13d3-4b51-b587-3240ce5b1064 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:49.840395 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-617bd881-13d3-4b51-b587-3240ce5b1064 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:49.843344 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1262/2503] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:52:49 2022] DELETE /identity/v3/projects/a1885f63b4014df7bc4d675070603132 => generated 0 bytes in 281 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:52:49.849244 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff931da-e3da-4d37-b148-e597db412ff7 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:49.849831 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff931da-e3da-4d37-b148-e597db412ff7 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:49.850098 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff931da-e3da-4d37-b148-e597db412ff7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:49.874976 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-4ff931da-e3da-4d37-b148-e597db412ff7 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:52:49.876240 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-4ff931da-e3da-4d37-b148-e597db412ff7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:49.941675 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:INSERT=2,keystone:DELETE=13,keystone:SELECT=85 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:52:49.984374 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1242/2504] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:52:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:52:49.998947 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a3bf60-74a9-4405-ba36-f4ae06dccc46 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:49.999955 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a3bf60-74a9-4405-ba36-f4ae06dccc46 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:50.000405 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a3bf60-74a9-4405-ba36-f4ae06dccc46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:50.037733 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-a0a3bf60-74a9-4405-ba36-f4ae06dccc46 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:52:50.040430 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a0a3bf60-74a9-4405-ba36-f4ae06dccc46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:50.056068 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff931da-e3da-4d37-b148-e597db412ff7 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:50.057530 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-4ff931da-e3da-4d37-b148-e597db412ff7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:50.065339 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1263/2505] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:52:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 70 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:52:50.077546 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0a3bf60-74a9-4405-ba36-f4ae06dccc46 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:50.084885 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff931da-e3da-4d37-b148-e597db412ff7 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:50.087822 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0a3bf60-74a9-4405-ba36-f4ae06dccc46 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:50.096810 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1827fee7-6672-49ba-b646-056a87c613f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:50.098087 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6442a949-971d-4d3f-a5f5-c16a77629798 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:50.098265 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1827fee7-6672-49ba-b646-056a87c613f4 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:50.098466 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6442a949-971d-4d3f-a5f5-c16a77629798 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:50.098660 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6442a949-971d-4d3f-a5f5-c16a77629798 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:50.098802 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1827fee7-6672-49ba-b646-056a87c613f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:50.099179 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1827fee7-6672-49ba-b646-056a87c613f4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:50.099351 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6442a949-971d-4d3f-a5f5-c16a77629798 admin admin] PATH_INFO: `/v3/projects/a61b4331030145e4a2b3b6697eb1b2ae` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:50.100356 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1827fee7-6672-49ba-b646-056a87c613f4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:52:50.100545 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6442a949-971d-4d3f-a5f5-c16a77629798 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a61b4331030145e4a2b3b6697eb1b2ae)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:50.101644 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1827fee7-6672-49ba-b646-056a87c613f4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:50.102609 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6442a949-971d-4d3f-a5f5-c16a77629798 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:50.104754 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1827fee7-6672-49ba-b646-056a87c613f4 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:50.130544 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1264/2506] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:52:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:52:50.135777 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1827fee7-6672-49ba-b646-056a87c613f4 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:50.157591 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1827fee7-6672-49ba-b646-056a87c613f4 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:50.176022 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-6442a949-971d-4d3f-a5f5-c16a77629798 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a61b4331030145e4a2b3b6697eb1b2ae', 'request_id': 'req-6442a949-971d-4d3f-a5f5-c16a77629798'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:50.184571 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-6442a949-971d-4d3f-a5f5-c16a77629798 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a61b4331030145e4a2b3b6697eb1b2ae', 'request_id': 'req-6442a949-971d-4d3f-a5f5-c16a77629798'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:50.195046 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-6442a949-971d-4d3f-a5f5-c16a77629798 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a61b4331030145e4a2b3b6697eb1b2ae', 'request_id': 'req-6442a949-971d-4d3f-a5f5-c16a77629798'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:50.222960 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-6442a949-971d-4d3f-a5f5-c16a77629798 admin admin] The token cache is being invalidate because project a61b4331030145e4a2b3b6697eb1b2ae was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:52:50.223854 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-6442a949-971d-4d3f-a5f5-c16a77629798 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:50.225558 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-6442a949-971d-4d3f-a5f5-c16a77629798 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:52:50.229445 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1243/2507] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:52:50 2022] DELETE /identity/v3/projects/a61b4331030145e4a2b3b6697eb1b2ae => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:52:50.239811 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83142a00-6b5e-4c36-bdb1-b23d154fa7cb None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:50.240536 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83142a00-6b5e-4c36-bdb1-b23d154fa7cb None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:50.241308 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83142a00-6b5e-4c36-bdb1-b23d154fa7cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:50.278777 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02cb1758-96f4-49ec-8d5a-c59f4be5a00c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:50.279934 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cb1758-96f4-49ec-8d5a-c59f4be5a00c admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:50.280414 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cb1758-96f4-49ec-8d5a-c59f4be5a00c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:50.280909 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cb1758-96f4-49ec-8d5a-c59f4be5a00c admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:50.281200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-83142a00-6b5e-4c36-bdb1-b23d154fa7cb None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:52:50.282147 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02cb1758-96f4-49ec-8d5a-c59f4be5a00c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:50.282796 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-83142a00-6b5e-4c36-bdb1-b23d154fa7cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:50.286019 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02cb1758-96f4-49ec-8d5a-c59f4be5a00c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:50.291945 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1244/2508] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:52:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 56 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:52:50.300635 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f6a74c-f29c-4d5b-8399-757934496ac8 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:50.301055 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f6a74c-f29c-4d5b-8399-757934496ac8 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:50.301341 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f6a74c-f29c-4d5b-8399-757934496ac8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:50.322602 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:52:50.327782 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-39f6a74c-f29c-4d5b-8399-757934496ac8 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:52:50.329559 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-39f6a74c-f29c-4d5b-8399-757934496ac8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:50.333824 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1265/2509] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:52:50 2022] POST /identity/v3/projects => generated 363 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:52:50.338759 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02cb1758-96f4-49ec-8d5a-c59f4be5a00c admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:50.341363 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-02cb1758-96f4-49ec-8d5a-c59f4be5a00c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:50.423359 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1245/2510] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:52:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:52:50.426785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02cb1758-96f4-49ec-8d5a-c59f4be5a00c admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:50.431108 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f6a74c-f29c-4d5b-8399-757934496ac8 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:50.432191 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50d1d73-3daa-4520-9316-40973312229a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:50.432827 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50d1d73-3daa-4520-9316-40973312229a admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:50.433118 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50d1d73-3daa-4520-9316-40973312229a admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:50.433388 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f50d1d73-3daa-4520-9316-40973312229a admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:50.434045 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f50d1d73-3daa-4520-9316-40973312229a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:50.436682 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f50d1d73-3daa-4520-9316-40973312229a admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:50.446306 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f6a74c-f29c-4d5b-8399-757934496ac8 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:50.453301 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2250068c-504c-440d-8e76-3824611c4f9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:50.454166 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2250068c-504c-440d-8e76-3824611c4f9e admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:50.454651 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2250068c-504c-440d-8e76-3824611c4f9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:50.455100 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2250068c-504c-440d-8e76-3824611c4f9e admin admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:50.455999 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2250068c-504c-440d-8e76-3824611c4f9e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:52:50.456381 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2250068c-504c-440d-8e76-3824611c4f9e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:50.458365 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2250068c-504c-440d-8e76-3824611c4f9e admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:50.461857 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1266/2511] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:52:50 2022] POST /identity/v3/users => generated 282 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:52:50.466325 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50d1d73-3daa-4520-9316-40973312229a admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:50.478494 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1246/2512] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:52:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:52:50.478802 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f50d1d73-3daa-4520-9316-40973312229a admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:50.479777 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2250068c-504c-440d-8e76-3824611c4f9e admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:50.483780 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49db2789-f654-4eb4-82d5-da6c267d6884 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:50.484424 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49db2789-f654-4eb4-82d5-da6c267d6884 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:50.484721 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49db2789-f654-4eb4-82d5-da6c267d6884 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:50.484984 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49db2789-f654-4eb4-82d5-da6c267d6884 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:50.485651 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49db2789-f654-4eb4-82d5-da6c267d6884 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:50.487341 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49db2789-f654-4eb4-82d5-da6c267d6884 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:50.492834 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2250068c-504c-440d-8e76-3824611c4f9e admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:50.499153 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6c8bcc-078d-41d5-8cac-28b303c01c1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:50.499631 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1267/2513] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:52:50 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:52:50.500077 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6c8bcc-078d-41d5-8cac-28b303c01c1e admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:50.500503 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6c8bcc-078d-41d5-8cac-28b303c01c1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:50.500851 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6c8bcc-078d-41d5-8cac-28b303c01c1e admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:50.501569 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed6c8bcc-078d-41d5-8cac-28b303c01c1e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:50.504309 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49db2789-f654-4eb4-82d5-da6c267d6884 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:50.504590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed6c8bcc-078d-41d5-8cac-28b303c01c1e admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:50.515370 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49db2789-f654-4eb4-82d5-da6c267d6884 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:50.521625 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6da8fea-bb12-4e8e-b8c2-3be13168ef9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:50.522261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6da8fea-bb12-4e8e-b8c2-3be13168ef9f admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:50.522578 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6da8fea-bb12-4e8e-b8c2-3be13168ef9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:50.522856 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6da8fea-bb12-4e8e-b8c2-3be13168ef9f admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:50.523518 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6da8fea-bb12-4e8e-b8c2-3be13168ef9f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:50.525171 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6da8fea-bb12-4e8e-b8c2-3be13168ef9f admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:50.538437 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1268/2514] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:52:50 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:52:50.543209 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1247/2515] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:52:50 2022] POST /identity/v3/projects => generated 409 bytes in 64 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:52:50.545327 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6da8fea-bb12-4e8e-b8c2-3be13168ef9f admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:50.549646 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6c8bcc-078d-41d5-8cac-28b303c01c1e admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:50.571064 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6da8fea-bb12-4e8e-b8c2-3be13168ef9f admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:50.595154 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:52:50.641954 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6c8bcc-078d-41d5-8cac-28b303c01c1e admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:50.647333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff50b50-5ecd-4a0e-bef8-912ae2b0d8f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:50.647964 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff50b50-5ecd-4a0e-bef8-912ae2b0d8f0 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:50.648359 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff50b50-5ecd-4a0e-bef8-912ae2b0d8f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:50.648637 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff50b50-5ecd-4a0e-bef8-912ae2b0d8f0 admin admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:50.649340 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ff50b50-5ecd-4a0e-bef8-912ae2b0d8f0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:50.651425 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ff50b50-5ecd-4a0e-bef8-912ae2b0d8f0 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:50.669299 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7eb414f-b123-4716-b345-f8194c0c27d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:50.670033 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7eb414f-b123-4716-b345-f8194c0c27d8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:50.670425 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7eb414f-b123-4716-b345-f8194c0c27d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:50.670968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7eb414f-b123-4716-b345-f8194c0c27d8 admin admin] PATH_INFO: `/v3/projects/87d9e193941c4afe8d81e422938ea996/users/75bedea0069b484d83f0f426fda05500/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:50.678711 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1248/2516] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:52:50 2022] POST /identity/v3/users => generated 305 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:52:50.683540 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff50b50-5ecd-4a0e-bef8-912ae2b0d8f0 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:50.690980 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7eb414f-b123-4716-b345-f8194c0c27d8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=87d9e193941c4afe8d81e422938ea996, user_id=75bedea0069b484d83f0f426fda05500, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:50.694088 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7eb414f-b123-4716-b345-f8194c0c27d8 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:50.695875 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff50b50-5ecd-4a0e-bef8-912ae2b0d8f0 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:50.702637 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a1436dc-4473-459b-9b68-7a2fb58493dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:50.704168 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1436dc-4473-459b-9b68-7a2fb58493dd admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:50.704740 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1436dc-4473-459b-9b68-7a2fb58493dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:50.705291 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1436dc-4473-459b-9b68-7a2fb58493dd admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:50.706784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a1436dc-4473-459b-9b68-7a2fb58493dd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:50.709656 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a1436dc-4473-459b-9b68-7a2fb58493dd admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:50.714641 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1269/2517] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:52:50 2022] PUT /identity/v3/projects/87d9e193941c4afe8d81e422938ea996/users/75bedea0069b484d83f0f426fda05500/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:52:50.729603 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1249/2518] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:52:50 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:52:50.731947 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20847003-381d-4040-a4fb-29dcd7457ac2 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:50.732469 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20847003-381d-4040-a4fb-29dcd7457ac2 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:50.732861 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20847003-381d-4040-a4fb-29dcd7457ac2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:50.735159 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a1436dc-4473-459b-9b68-7a2fb58493dd admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:50.751927 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a1436dc-4473-459b-9b68-7a2fb58493dd admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:50.776813 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-20847003-381d-4040-a4fb-29dcd7457ac2 None None] MFA Rules not processed for user `75bedea0069b484d83f0f426fda05500`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:52:50.778144 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-20847003-381d-4040-a4fb-29dcd7457ac2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:50.844892 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8902b160-cd80-42c9-a07d-ad0d7da88f7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:50.845778 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8902b160-cd80-42c9-a07d-ad0d7da88f7c admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:50.846034 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:52:50.846238 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8902b160-cd80-42c9-a07d-ad0d7da88f7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:50.846999 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8902b160-cd80-42c9-a07d-ad0d7da88f7c admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:50.848215 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8902b160-cd80-42c9-a07d-ad0d7da88f7c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:50.851237 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8902b160-cd80-42c9-a07d-ad0d7da88f7c admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:50.870182 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1250/2519] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:52:50 2022] GET /identity/v3/roles => generated 1420 bytes in 131 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:52:50.872256 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8902b160-cd80-42c9-a07d-ad0d7da88f7c admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:50.883393 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1270/2520] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:52:50 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:52:50.888455 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8902b160-cd80-42c9-a07d-ad0d7da88f7c admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:50.889916 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20847003-381d-4040-a4fb-29dcd7457ac2 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:50.895467 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57781cf-b3b8-4fda-a094-03e425604635 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:50.896288 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57781cf-b3b8-4fda-a094-03e425604635 admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:50.896692 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57781cf-b3b8-4fda-a094-03e425604635 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:50.897067 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57781cf-b3b8-4fda-a094-03e425604635 admin admin] PATH_INFO: `/v3/projects/6e0572ae10cb4b979381039123dd97ad/users/dc716c71ccbf46f1be08597850ce8634/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:50.909227 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20847003-381d-4040-a4fb-29dcd7457ac2 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:50.916871 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a57781cf-b3b8-4fda-a094-03e425604635 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6e0572ae10cb4b979381039123dd97ad, user_id=dc716c71ccbf46f1be08597850ce8634, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:50.917566 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0fcfd4-166f-4756-a549-3d519d4986b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:50.918652 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0fcfd4-166f-4756-a549-3d519d4986b7 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:50.919166 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0fcfd4-166f-4756-a549-3d519d4986b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:50.919830 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0fcfd4-166f-4756-a549-3d519d4986b7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:50.919951 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a57781cf-b3b8-4fda-a094-03e425604635 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:50.921011 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b0fcfd4-166f-4756-a549-3d519d4986b7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:50.925105 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b0fcfd4-166f-4756-a549-3d519d4986b7 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:50.937389 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1251/2521] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:52:50 2022] PUT /identity/v3/projects/6e0572ae10cb4b979381039123dd97ad/users/dc716c71ccbf46f1be08597850ce8634/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:52:50.948597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e62bea-06d8-4438-9140-efd74577c495 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:50.948888 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e62bea-06d8-4438-9140-efd74577c495 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:50.949160 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e62bea-06d8-4438-9140-efd74577c495 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:50.972694 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1271/2522] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:52:50 2022] POST /identity/v3/projects => generated 363 bytes in 84 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:52:50.977277 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0fcfd4-166f-4756-a549-3d519d4986b7 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:50.984061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-88e62bea-06d8-4438-9140-efd74577c495 None None] MFA Rules not processed for user `dc716c71ccbf46f1be08597850ce8634`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:52:50.986267 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-88e62bea-06d8-4438-9140-efd74577c495 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:51.077696 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1252/2523] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:52:50 2022] POST /identity/v3/auth/tokens => generated 3286 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:52:51.084543 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88e62bea-06d8-4438-9140-efd74577c495 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:51.085325 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0fcfd4-166f-4756-a549-3d519d4986b7 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:51.090886 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e574eae-f10a-4c8d-8c12-258ad23324c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:51.091513 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e574eae-f10a-4c8d-8c12-258ad23324c3 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:51.091790 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e574eae-f10a-4c8d-8c12-258ad23324c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:51.092066 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e574eae-f10a-4c8d-8c12-258ad23324c3 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:51.092722 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e574eae-f10a-4c8d-8c12-258ad23324c3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:51.094746 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e574eae-f10a-4c8d-8c12-258ad23324c3 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:51.099412 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88e62bea-06d8-4438-9140-efd74577c495 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:51.104889 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c54dbfa-f70e-4b85-b61b-2fe3b5a52314 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:51.105639 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c54dbfa-f70e-4b85-b61b-2fe3b5a52314 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:51.106015 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c54dbfa-f70e-4b85-b61b-2fe3b5a52314 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:51.106381 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c54dbfa-f70e-4b85-b61b-2fe3b5a52314 admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:51.107192 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c54dbfa-f70e-4b85-b61b-2fe3b5a52314 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:51.111444 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c54dbfa-f70e-4b85-b61b-2fe3b5a52314 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:51.118597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:52:51.122450 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1272/2524] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:52:50 2022] POST /identity/v3/users => generated 288 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:52:51.128122 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e574eae-f10a-4c8d-8c12-258ad23324c3 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:51.147218 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e574eae-f10a-4c8d-8c12-258ad23324c3 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:51.156774 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f420bc5-d7a3-49a5-9c7a-528e27b12228 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:51.157469 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f420bc5-d7a3-49a5-9c7a-528e27b12228 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:51.157789 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f420bc5-d7a3-49a5-9c7a-528e27b12228 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:51.158063 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f420bc5-d7a3-49a5-9c7a-528e27b12228 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:51.158913 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f420bc5-d7a3-49a5-9c7a-528e27b12228 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:51.165382 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1253/2525] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:52:51 2022] POST /identity/v3/projects => generated 409 bytes in 77 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:52:51.165662 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f420bc5-d7a3-49a5-9c7a-528e27b12228 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:51.167254 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c54dbfa-f70e-4b85-b61b-2fe3b5a52314 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:51.176965 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1273/2526] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:52:51 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:52:51.182033 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f420bc5-d7a3-49a5-9c7a-528e27b12228 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:51.194327 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f420bc5-d7a3-49a5-9c7a-528e27b12228 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:51.262436 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c54dbfa-f70e-4b85-b61b-2fe3b5a52314 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:51.268798 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf5e3250-4e52-440c-bc4f-6ffd66880041 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:51.269798 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5e3250-4e52-440c-bc4f-6ffd66880041 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:51.270387 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5e3250-4e52-440c-bc4f-6ffd66880041 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:51.270903 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf5e3250-4e52-440c-bc4f-6ffd66880041 admin admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:51.272296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf5e3250-4e52-440c-bc4f-6ffd66880041 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:51.274931 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf5e3250-4e52-440c-bc4f-6ffd66880041 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:51.276472 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc8db21c-baea-410d-b76c-d386aa2befb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:51.277481 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8db21c-baea-410d-b76c-d386aa2befb9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:51.277891 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8db21c-baea-410d-b76c-d386aa2befb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:51.278271 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8db21c-baea-410d-b76c-d386aa2befb9 admin admin] PATH_INFO: `/v3/projects/d5934f936f52409f9fba8c7c28d3a32d/users/0fb9a0a552ae468dbc3935d515c63556/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:51.307083 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc8db21c-baea-410d-b76c-d386aa2befb9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d5934f936f52409f9fba8c7c28d3a32d, user_id=0fb9a0a552ae468dbc3935d515c63556, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:51.310302 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1254/2527] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:52:51 2022] POST /identity/v3/users => generated 311 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:52:51.311554 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc8db21c-baea-410d-b76c-d386aa2befb9 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:51.315363 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf5e3250-4e52-440c-bc4f-6ffd66880041 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:51.333692 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf5e3250-4e52-440c-bc4f-6ffd66880041 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:51.334126 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1274/2528] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:52:51 2022] PUT /identity/v3/projects/d5934f936f52409f9fba8c7c28d3a32d/users/0fb9a0a552ae468dbc3935d515c63556/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:52:51.339646 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc8db21c-baea-410d-b76c-d386aa2befb9 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:51.376446 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:52:51.446053 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d85d748d-0b47-405d-9e75-cbed88740b71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:51.447188 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85d748d-0b47-405d-9e75-cbed88740b71 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:51.447866 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85d748d-0b47-405d-9e75-cbed88740b71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:51.448386 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85d748d-0b47-405d-9e75-cbed88740b71 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:51.449358 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d85d748d-0b47-405d-9e75-cbed88740b71 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:51.452616 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d85d748d-0b47-405d-9e75-cbed88740b71 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:51.455670 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc8db21c-baea-410d-b76c-d386aa2befb9 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:51.463976 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1bb6894-a6c1-4165-adb5-cc577b8c62d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:51.464989 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1bb6894-a6c1-4165-adb5-cc577b8c62d8 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:51.465462 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1bb6894-a6c1-4165-adb5-cc577b8c62d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:51.465932 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1bb6894-a6c1-4165-adb5-cc577b8c62d8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:51.467811 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1bb6894-a6c1-4165-adb5-cc577b8c62d8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:51.471014 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1bb6894-a6c1-4165-adb5-cc577b8c62d8 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:51.476136 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1255/2529] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:52:51 2022] GET /identity/v3/roles => generated 1420 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:52:51.480559 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d85d748d-0b47-405d-9e75-cbed88740b71 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:51.492973 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1275/2530] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:52:51 2022] GET /identity/v3/roles => generated 1420 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:52:51.496865 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1bb6894-a6c1-4165-adb5-cc577b8c62d8 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:51.498993 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d85d748d-0b47-405d-9e75-cbed88740b71 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:51.507474 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e15ad3a0-2eb4-452f-a2da-832888e44844 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:51.508602 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15ad3a0-2eb4-452f-a2da-832888e44844 admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:51.509072 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15ad3a0-2eb4-452f-a2da-832888e44844 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:51.509535 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e15ad3a0-2eb4-452f-a2da-832888e44844 admin admin] PATH_INFO: `/v3/projects/b3ed0472da894cff904311d71cb40088/users/c51791cda31441b39f5003b84f84158b/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:51.511200 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1bb6894-a6c1-4165-adb5-cc577b8c62d8 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:51.518202 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8265aa0f-430b-4a01-a931-6ed4c7309945 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:51.519268 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8265aa0f-430b-4a01-a931-6ed4c7309945 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:51.519713 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8265aa0f-430b-4a01-a931-6ed4c7309945 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:51.520156 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8265aa0f-430b-4a01-a931-6ed4c7309945 admin admin] PATH_INFO: `/v3/projects/d5934f936f52409f9fba8c7c28d3a32d/users/0fb9a0a552ae468dbc3935d515c63556/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:51.524063 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8265aa0f-430b-4a01-a931-6ed4c7309945 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d5934f936f52409f9fba8c7c28d3a32d, user_id=0fb9a0a552ae468dbc3935d515c63556, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:51.528483 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8265aa0f-430b-4a01-a931-6ed4c7309945 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:51.536344 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e15ad3a0-2eb4-452f-a2da-832888e44844 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b3ed0472da894cff904311d71cb40088, user_id=c51791cda31441b39f5003b84f84158b, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:51.539149 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e15ad3a0-2eb4-452f-a2da-832888e44844 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:51.549503 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1276/2531] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:52:51 2022] PUT /identity/v3/projects/d5934f936f52409f9fba8c7c28d3a32d/users/0fb9a0a552ae468dbc3935d515c63556/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:52:51.564082 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1256/2532] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:52:51 2022] PUT /identity/v3/projects/b3ed0472da894cff904311d71cb40088/users/c51791cda31441b39f5003b84f84158b/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 84 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:52:51.568620 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b20f731-b65e-45ad-9370-2035a5061799 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:51.568885 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e15ad3a0-2eb4-452f-a2da-832888e44844 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:51.569059 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b20f731-b65e-45ad-9370-2035a5061799 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:51.574502 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b20f731-b65e-45ad-9370-2035a5061799 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:51.600773 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-5b20f731-b65e-45ad-9370-2035a5061799 None None] MFA Rules not processed for user `0fb9a0a552ae468dbc3935d515c63556`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:52:51.602018 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-5b20f731-b65e-45ad-9370-2035a5061799 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:51.646191 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:52:51.664716 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e15ad3a0-2eb4-452f-a2da-832888e44844 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:51.669800 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac18646-7763-49ce-8495-765bb9db8895 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:51.671037 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac18646-7763-49ce-8495-765bb9db8895 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:51.671318 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac18646-7763-49ce-8495-765bb9db8895 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:51.671905 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ac18646-7763-49ce-8495-765bb9db8895 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:51.672499 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac18646-7763-49ce-8495-765bb9db8895 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:51.674131 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ac18646-7763-49ce-8495-765bb9db8895 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:51.685598 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1257/2533] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:52:51 2022] GET /identity/v3/roles => generated 1420 bytes in 118 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:52:51.690145 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac18646-7763-49ce-8495-765bb9db8895 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:51.700141 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1277/2534] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:52:51 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:52:51.701578 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ac18646-7763-49ce-8495-765bb9db8895 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:51.707026 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2820ddd8-7131-4d5c-988c-813d4659e934 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:51.707720 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2820ddd8-7131-4d5c-988c-813d4659e934 admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:51.708002 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2820ddd8-7131-4d5c-988c-813d4659e934 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:51.708283 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2820ddd8-7131-4d5c-988c-813d4659e934 admin admin] PATH_INFO: `/v3/projects/b3ed0472da894cff904311d71cb40088/users/c51791cda31441b39f5003b84f84158b/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:51.711466 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2820ddd8-7131-4d5c-988c-813d4659e934 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b3ed0472da894cff904311d71cb40088, user_id=c51791cda31441b39f5003b84f84158b, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:51.714185 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2820ddd8-7131-4d5c-988c-813d4659e934 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:51.720460 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b20f731-b65e-45ad-9370-2035a5061799 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:51.722812 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-5b20f731-b65e-45ad-9370-2035a5061799 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:51.727808 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1258/2535] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:52:51 2022] PUT /identity/v3/projects/b3ed0472da894cff904311d71cb40088/users/c51791cda31441b39f5003b84f84158b/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:52:51.739914 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7ea068-602a-46fb-85cf-e884e5c834f1 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:51.740203 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7ea068-602a-46fb-85cf-e884e5c834f1 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:51.740456 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7ea068-602a-46fb-85cf-e884e5c834f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:51.772037 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-3b7ea068-602a-46fb-85cf-e884e5c834f1 None None] MFA Rules not processed for user `c51791cda31441b39f5003b84f84158b`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:52:51.773404 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-3b7ea068-602a-46fb-85cf-e884e5c834f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:51.826896 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b20f731-b65e-45ad-9370-2035a5061799 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:51.873947 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1259/2536] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:52:51 2022] POST /identity/v3/auth/tokens => generated 3353 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:52:51.880817 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:52:51.892672 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7ea068-602a-46fb-85cf-e884e5c834f1 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:51.892965 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53110ab1-fe21-4ce8-bf17-d32968274b36 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:51.893716 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53110ab1-fe21-4ce8-bf17-d32968274b36 service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:51.894202 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53110ab1-fe21-4ce8-bf17-d32968274b36 service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:51.894747 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53110ab1-fe21-4ce8-bf17-d32968274b36 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:51.894943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-3b7ea068-602a-46fb-85cf-e884e5c834f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:52:51.914322 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7ea068-602a-46fb-85cf-e884e5c834f1 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:51.920419 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ceffd62-81de-4305-adba-ccafd5e99d0a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:51.921379 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ceffd62-81de-4305-adba-ccafd5e99d0a service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:51.922097 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ceffd62-81de-4305-adba-ccafd5e99d0a service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:51.922590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ceffd62-81de-4305-adba-ccafd5e99d0a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:51.937242 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ceffd62-81de-4305-adba-ccafd5e99d0a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:51.939264 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ceffd62-81de-4305-adba-ccafd5e99d0a service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:51.953268 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1260/2537] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:52:51 2022] GET /identity/v3/auth/tokens => generated 3353 bytes in 61 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:52:51.996273 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53110ab1-fe21-4ce8-bf17-d32968274b36 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:51.999344 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53110ab1-fe21-4ce8-bf17-d32968274b36 service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:52.019630 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1278/2538] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:52:51 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 300 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:52:52.124929 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ceffd62-81de-4305-adba-ccafd5e99d0a service cinder] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:52.136081 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ceffd62-81de-4305-adba-ccafd5e99d0a service cinder] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:52.141246 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d844f4-2c56-4ce7-84bf-de21747d6c73 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:52.141868 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d844f4-2c56-4ce7-84bf-de21747d6c73 service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:52.142146 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d844f4-2c56-4ce7-84bf-de21747d6c73 service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:52.142477 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d844f4-2c56-4ce7-84bf-de21747d6c73 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:52.193484 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53110ab1-fe21-4ce8-bf17-d32968274b36 service cinder] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:52:52.203373 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53110ab1-fe21-4ce8-bf17-d32968274b36 service cinder] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:52:52.207299 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c9369ca-59db-40ef-ba6a-5cc70a7e87cf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:52:52.207922 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9369ca-59db-40ef-ba6a-5cc70a7e87cf service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:52:52.208202 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9369ca-59db-40ef-ba6a-5cc70a7e87cf service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:52:52.208475 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9369ca-59db-40ef-ba6a-5cc70a7e87cf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:52:52.228645 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d844f4-2c56-4ce7-84bf-de21747d6c73 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:52.231481 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d844f4-2c56-4ce7-84bf-de21747d6c73 service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:52.246721 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1261/2539] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:52:52 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:52:52.286421 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c9369ca-59db-40ef-ba6a-5cc70a7e87cf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:52:52.287664 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c9369ca-59db-40ef-ba6a-5cc70a7e87cf service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:52:52.306812 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1279/2540] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:52:52 2022] GET /identity/v3/auth/tokens => generated 3286 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:53:02.241419 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=81,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:02.293980 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=58 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:02.312999 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d844f4-2c56-4ce7-84bf-de21747d6c73 service cinder] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:02.499649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d844f4-2c56-4ce7-84bf-de21747d6c73 service cinder] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:02.508959 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2031491a-1b90-4a6f-8aa5-d65a6b62a556 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:02.510477 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2031491a-1b90-4a6f-8aa5-d65a6b62a556 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:02.511148 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2031491a-1b90-4a6f-8aa5-d65a6b62a556 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:02.511626 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2031491a-1b90-4a6f-8aa5-d65a6b62a556 admin admin] PATH_INFO: `/v3/users/dc716c71ccbf46f1be08597850ce8634` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:02.513130 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2031491a-1b90-4a6f-8aa5-d65a6b62a556 admin admin] RBAC: Authorizing `identity:delete_user(user_id=dc716c71ccbf46f1be08597850ce8634)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:02.515816 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2031491a-1b90-4a6f-8aa5-d65a6b62a556 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:02.596643 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-2031491a-1b90-4a6f-8aa5-d65a6b62a556 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dc716c71ccbf46f1be08597850ce8634'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:02.610435 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-2031491a-1b90-4a6f-8aa5-d65a6b62a556 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dc716c71ccbf46f1be08597850ce8634'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:02.611861 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-2031491a-1b90-4a6f-8aa5-d65a6b62a556 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dc716c71ccbf46f1be08597850ce8634'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:02.613117 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-2031491a-1b90-4a6f-8aa5-d65a6b62a556 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dc716c71ccbf46f1be08597850ce8634'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:02.636205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-2031491a-1b90-4a6f-8aa5-d65a6b62a556 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dc716c71ccbf46f1be08597850ce8634'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:02.686200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1262/2541] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:53:02 2022] DELETE /identity/v3/users/dc716c71ccbf46f1be08597850ce8634 => generated 0 bytes in 375 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:02.694785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce113d09-7bf1-4888-b6af-79e208be714d None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:02.695539 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce113d09-7bf1-4888-b6af-79e208be714d None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:02.696146 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce113d09-7bf1-4888-b6af-79e208be714d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:02.810078 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-ce113d09-7bf1-4888-b6af-79e208be714d None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:02.812448 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ce113d09-7bf1-4888-b6af-79e208be714d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:02.951959 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1280/2542] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 261 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:53:02.965509 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2031491a-1b90-4a6f-8aa5-d65a6b62a556 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:02.968260 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-2031491a-1b90-4a6f-8aa5-d65a6b62a556 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:03.059219 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:03.113305 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2031491a-1b90-4a6f-8aa5-d65a6b62a556 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:03.117902 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb0465c5-9da2-4099-bee2-0b8547addfb8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:03.118701 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0465c5-9da2-4099-bee2-0b8547addfb8 service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:03.119001 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0465c5-9da2-4099-bee2-0b8547addfb8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:03.119390 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb0465c5-9da2-4099-bee2-0b8547addfb8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:03.134457 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb0465c5-9da2-4099-bee2-0b8547addfb8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:03.135111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb0465c5-9da2-4099-bee2-0b8547addfb8 service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:03.148617 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1263/2543] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:53:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:53:03.611732 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce113d09-7bf1-4888-b6af-79e208be714d None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:03.613882 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ce113d09-7bf1-4888-b6af-79e208be714d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:03.637883 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce113d09-7bf1-4888-b6af-79e208be714d None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:03.646844 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7270a0b6-ea78-45ae-ae55-177d6343e886 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:03.647892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7270a0b6-ea78-45ae-ae55-177d6343e886 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:03.648368 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7270a0b6-ea78-45ae-ae55-177d6343e886 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:03.648817 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7270a0b6-ea78-45ae-ae55-177d6343e886 admin admin] PATH_INFO: `/v3/projects/6e0572ae10cb4b979381039123dd97ad` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:03.650783 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7270a0b6-ea78-45ae-ae55-177d6343e886 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6e0572ae10cb4b979381039123dd97ad)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:03.654466 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7270a0b6-ea78-45ae-ae55-177d6343e886 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:03.722689 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7270a0b6-ea78-45ae-ae55-177d6343e886 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6e0572ae10cb4b979381039123dd97ad', 'request_id': 'req-7270a0b6-ea78-45ae-ae55-177d6343e886'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:03.730867 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7270a0b6-ea78-45ae-ae55-177d6343e886 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6e0572ae10cb4b979381039123dd97ad', 'request_id': 'req-7270a0b6-ea78-45ae-ae55-177d6343e886'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:03.741330 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7270a0b6-ea78-45ae-ae55-177d6343e886 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6e0572ae10cb4b979381039123dd97ad', 'request_id': 'req-7270a0b6-ea78-45ae-ae55-177d6343e886'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:03.755033 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7270a0b6-ea78-45ae-ae55-177d6343e886 admin admin] The token cache is being invalidate because project 6e0572ae10cb4b979381039123dd97ad was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:53:03.755463 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7270a0b6-ea78-45ae-ae55-177d6343e886 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:03.756399 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7270a0b6-ea78-45ae-ae55-177d6343e886 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:03.758241 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1281/2544] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:53:03 2022] DELETE /identity/v3/projects/6e0572ae10cb4b979381039123dd97ad => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:03.763108 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb0465c5-9da2-4099-bee2-0b8547addfb8 service neutron] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:03.780674 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb0465c5-9da2-4099-bee2-0b8547addfb8 service neutron] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:03.883079 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621851cf-241b-4699-ab7e-36ec02d2665b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:03.884016 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621851cf-241b-4699-ab7e-36ec02d2665b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:03.884450 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621851cf-241b-4699-ab7e-36ec02d2665b admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:03.884902 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621851cf-241b-4699-ab7e-36ec02d2665b admin admin] PATH_INFO: `/v3/users/c51791cda31441b39f5003b84f84158b` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:03.887102 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621851cf-241b-4699-ab7e-36ec02d2665b admin admin] RBAC: Authorizing `identity:delete_user(user_id=c51791cda31441b39f5003b84f84158b)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:03.890529 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621851cf-241b-4699-ab7e-36ec02d2665b admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:03.968601 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-621851cf-241b-4699-ab7e-36ec02d2665b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c51791cda31441b39f5003b84f84158b'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:03.980345 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-621851cf-241b-4699-ab7e-36ec02d2665b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c51791cda31441b39f5003b84f84158b'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:03.981866 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-621851cf-241b-4699-ab7e-36ec02d2665b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c51791cda31441b39f5003b84f84158b'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:03.983592 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-621851cf-241b-4699-ab7e-36ec02d2665b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c51791cda31441b39f5003b84f84158b'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:03.997190 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-621851cf-241b-4699-ab7e-36ec02d2665b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c51791cda31441b39f5003b84f84158b'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:04.022991 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1264/2545] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:53:03 2022] DELETE /identity/v3/users/c51791cda31441b39f5003b84f84158b => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:53:04.702138 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7270a0b6-ea78-45ae-ae55-177d6343e886 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:04.704023 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7270a0b6-ea78-45ae-ae55-177d6343e886 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:04.723627 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621851cf-241b-4699-ab7e-36ec02d2665b admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:04.725578 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-621851cf-241b-4699-ab7e-36ec02d2665b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:04.726045 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7270a0b6-ea78-45ae-ae55-177d6343e886 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:04.812332 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:04.821986 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:04.832992 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b36ad2-a48e-42ad-9ba7-605697520c59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:04.834379 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b36ad2-a48e-42ad-9ba7-605697520c59 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:04.834908 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b36ad2-a48e-42ad-9ba7-605697520c59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:04.835460 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b36ad2-a48e-42ad-9ba7-605697520c59 admin admin] PATH_INFO: `/v3/users/75bedea0069b484d83f0f426fda05500` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:04.837310 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b36ad2-a48e-42ad-9ba7-605697520c59 admin admin] RBAC: Authorizing `identity:delete_user(user_id=75bedea0069b484d83f0f426fda05500)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:04.840186 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b36ad2-a48e-42ad-9ba7-605697520c59 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:04.846546 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621851cf-241b-4699-ab7e-36ec02d2665b admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:04.854043 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39ac2437-db98-4d31-8823-51e163114f6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:04.855066 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ac2437-db98-4d31-8823-51e163114f6c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:04.855520 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ac2437-db98-4d31-8823-51e163114f6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:04.855968 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ac2437-db98-4d31-8823-51e163114f6c admin admin] PATH_INFO: `/v3/projects/b3ed0472da894cff904311d71cb40088` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:04.857911 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39ac2437-db98-4d31-8823-51e163114f6c admin admin] RBAC: Authorizing `identity:delete_project(project_id=b3ed0472da894cff904311d71cb40088)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:04.861050 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39ac2437-db98-4d31-8823-51e163114f6c admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:04.916267 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a5b36ad2-a48e-42ad-9ba7-605697520c59 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '75bedea0069b484d83f0f426fda05500'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:04.930506 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a5b36ad2-a48e-42ad-9ba7-605697520c59 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '75bedea0069b484d83f0f426fda05500'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:04.931695 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a5b36ad2-a48e-42ad-9ba7-605697520c59 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '75bedea0069b484d83f0f426fda05500'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:04.932768 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a5b36ad2-a48e-42ad-9ba7-605697520c59 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '75bedea0069b484d83f0f426fda05500'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:04.943460 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-39ac2437-db98-4d31-8823-51e163114f6c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b3ed0472da894cff904311d71cb40088', 'request_id': 'req-39ac2437-db98-4d31-8823-51e163114f6c'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:04.945551 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a5b36ad2-a48e-42ad-9ba7-605697520c59 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '75bedea0069b484d83f0f426fda05500'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:04.962494 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-39ac2437-db98-4d31-8823-51e163114f6c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b3ed0472da894cff904311d71cb40088', 'request_id': 'req-39ac2437-db98-4d31-8823-51e163114f6c'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:04.984439 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-39ac2437-db98-4d31-8823-51e163114f6c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b3ed0472da894cff904311d71cb40088', 'request_id': 'req-39ac2437-db98-4d31-8823-51e163114f6c'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:04.988200 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1282/2546] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:53:04 2022] DELETE /identity/v3/users/75bedea0069b484d83f0f426fda05500 => generated 0 bytes in 287 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:04.994801 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b0bd60-d140-4043-837c-5d7da6723e10 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:04.995363 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b0bd60-d140-4043-837c-5d7da6723e10 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:04.995901 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9b0bd60-d140-4043-837c-5d7da6723e10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:05.015173 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-39ac2437-db98-4d31-8823-51e163114f6c admin admin] The token cache is being invalidate because project b3ed0472da894cff904311d71cb40088 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:53:05.015647 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-39ac2437-db98-4d31-8823-51e163114f6c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:05.016759 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-39ac2437-db98-4d31-8823-51e163114f6c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:05.019466 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1265/2547] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:53:04 2022] DELETE /identity/v3/projects/b3ed0472da894cff904311d71cb40088 => generated 0 bytes in 297 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:05.025819 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42141650-05e1-4f3f-a975-5e14b5c28e9f None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:05.026193 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42141650-05e1-4f3f-a975-5e14b5c28e9f None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:05.026557 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42141650-05e1-4f3f-a975-5e14b5c28e9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:05.030582 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-f9b0bd60-d140-4043-837c-5d7da6723e10 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:05.032399 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-f9b0bd60-d140-4043-837c-5d7da6723e10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:05.086206 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-42141650-05e1-4f3f-a975-5e14b5c28e9f None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:05.087530 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-42141650-05e1-4f3f-a975-5e14b5c28e9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:05.125282 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1283/2548] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:53:05.136445 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9b0bd60-d140-4043-837c-5d7da6723e10 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:05.137914 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-f9b0bd60-d140-4043-837c-5d7da6723e10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:05.169956 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1266/2549] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:53:05.179070 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51790086-9c31-47a9-a468-f3c6f358e677 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:05.179403 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51790086-9c31-47a9-a468-f3c6f358e677 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:05.179860 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51790086-9c31-47a9-a468-f3c6f358e677 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:05.207856 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-51790086-9c31-47a9-a468-f3c6f358e677 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:05.209707 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-51790086-9c31-47a9-a468-f3c6f358e677 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:05.224218 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1267/2550] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:53:05.224507 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51790086-9c31-47a9-a468-f3c6f358e677 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:05.236493 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51790086-9c31-47a9-a468-f3c6f358e677 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:05.236723 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9b0bd60-d140-4043-837c-5d7da6723e10 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:05.241814 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6483430-ad96-49eb-9cff-f8053066cd95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:05.243502 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a1cc1f-f7e8-45b0-b78e-6736a64a5b21 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:05.243723 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6483430-ad96-49eb-9cff-f8053066cd95 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:05.244503 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6483430-ad96-49eb-9cff-f8053066cd95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:05.244889 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a1cc1f-f7e8-45b0-b78e-6736a64a5b21 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:05.245242 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6483430-ad96-49eb-9cff-f8053066cd95 admin admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:05.245409 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a1cc1f-f7e8-45b0-b78e-6736a64a5b21 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:05.245965 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a1cc1f-f7e8-45b0-b78e-6736a64a5b21 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:05.246939 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6483430-ad96-49eb-9cff-f8053066cd95 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:53:05.247634 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6483430-ad96-49eb-9cff-f8053066cd95 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:05.251710 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6483430-ad96-49eb-9cff-f8053066cd95 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:05.267085 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a1cc1f-f7e8-45b0-b78e-6736a64a5b21 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:05.268785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9a1cc1f-f7e8-45b0-b78e-6736a64a5b21 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:05.280394 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1268/2551] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:53:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:53:05.284632 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1284/2552] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:53:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:53:05.286635 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6483430-ad96-49eb-9cff-f8053066cd95 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:05.306918 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6483430-ad96-49eb-9cff-f8053066cd95 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:05.315928 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ede438-3749-40d5-8238-8e777ace0cc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:05.317006 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ede438-3749-40d5-8238-8e777ace0cc9 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:05.317521 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ede438-3749-40d5-8238-8e777ace0cc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:05.318006 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ede438-3749-40d5-8238-8e777ace0cc9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:05.319265 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ede438-3749-40d5-8238-8e777ace0cc9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:05.323158 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ede438-3749-40d5-8238-8e777ace0cc9 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:05.378458 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1269/2553] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:53:05 2022] POST /identity/v3/projects => generated 371 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:53:05.383320 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a1cc1f-f7e8-45b0-b78e-6736a64a5b21 service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:05.395349 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a1cc1f-f7e8-45b0-b78e-6736a64a5b21 service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:05.421991 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:05.494463 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b79aea7d-134d-436f-a426-b92f5e059109 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:05.495271 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b79aea7d-134d-436f-a426-b92f5e059109 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:05.495718 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b79aea7d-134d-436f-a426-b92f5e059109 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:05.497059 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b79aea7d-134d-436f-a426-b92f5e059109 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:05.497470 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b79aea7d-134d-436f-a426-b92f5e059109 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:05.503087 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b79aea7d-134d-436f-a426-b92f5e059109 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:05.533421 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1285/2554] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:53:05 2022] POST /identity/v3/users => generated 286 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:53:05.538882 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ede438-3749-40d5-8238-8e777ace0cc9 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:05.554024 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ede438-3749-40d5-8238-8e777ace0cc9 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:05.558705 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfbaf5e0-00f9-4ccd-9f80-8f90f671e3b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:05.559384 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfbaf5e0-00f9-4ccd-9f80-8f90f671e3b8 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:05.559765 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfbaf5e0-00f9-4ccd-9f80-8f90f671e3b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:05.560007 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfbaf5e0-00f9-4ccd-9f80-8f90f671e3b8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:05.561780 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfbaf5e0-00f9-4ccd-9f80-8f90f671e3b8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:05.563810 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfbaf5e0-00f9-4ccd-9f80-8f90f671e3b8 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:05.567366 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:05.586210 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1270/2555] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:53:05 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:53:05.588001 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b79aea7d-134d-436f-a426-b92f5e059109 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:05.604909 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b79aea7d-134d-436f-a426-b92f5e059109 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:05.612306 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f17212d-e818-4ae0-90bf-d3adb8aa8ae6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:05.612833 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f17212d-e818-4ae0-90bf-d3adb8aa8ae6 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:05.613201 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f17212d-e818-4ae0-90bf-d3adb8aa8ae6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:05.613718 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f17212d-e818-4ae0-90bf-d3adb8aa8ae6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:05.614637 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f17212d-e818-4ae0-90bf-d3adb8aa8ae6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:05.616746 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f17212d-e818-4ae0-90bf-d3adb8aa8ae6 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:05.629582 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1286/2556] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:53:05 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:53:05.635245 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfbaf5e0-00f9-4ccd-9f80-8f90f671e3b8 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:05.654098 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfbaf5e0-00f9-4ccd-9f80-8f90f671e3b8 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:05.663573 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be45025b-5af5-4a62-b3c1-0f2b0630be79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:05.664277 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be45025b-5af5-4a62-b3c1-0f2b0630be79 admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:05.664551 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be45025b-5af5-4a62-b3c1-0f2b0630be79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:05.664835 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be45025b-5af5-4a62-b3c1-0f2b0630be79 admin admin] PATH_INFO: `/v3/projects/723d60ae52dc4b76ac83b79e46d8950f/users/03af8e89b8434a038fe593252633de19/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:05.697860 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be45025b-5af5-4a62-b3c1-0f2b0630be79 admin admin] RBAC: Authorizing `identity:create_grant(project_id=723d60ae52dc4b76ac83b79e46d8950f, user_id=03af8e89b8434a038fe593252633de19, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:05.703751 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be45025b-5af5-4a62-b3c1-0f2b0630be79 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:05.727866 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1271/2557] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:53:05 2022] PUT /identity/v3/projects/723d60ae52dc4b76ac83b79e46d8950f/users/03af8e89b8434a038fe593252633de19/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:05.735000 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3474bc-33dd-4dcc-a0b8-f7cddc2915ed None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:05.736039 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3474bc-33dd-4dcc-a0b8-f7cddc2915ed None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:05.736311 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3474bc-33dd-4dcc-a0b8-f7cddc2915ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:05.779518 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be45025b-5af5-4a62-b3c1-0f2b0630be79 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:05.782759 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-be45025b-5af5-4a62-b3c1-0f2b0630be79 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:05.787615 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-7e3474bc-33dd-4dcc-a0b8-f7cddc2915ed None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:05.789648 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7e3474bc-33dd-4dcc-a0b8-f7cddc2915ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:05.879157 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1287/2558] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:53:05.880700 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3474bc-33dd-4dcc-a0b8-f7cddc2915ed None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:05.926861 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be45025b-5af5-4a62-b3c1-0f2b0630be79 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:05.935523 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41a47e04-4078-4b06-931d-09150abf3d8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:05.936586 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a47e04-4078-4b06-931d-09150abf3d8b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:05.937076 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a47e04-4078-4b06-931d-09150abf3d8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:05.937553 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a47e04-4078-4b06-931d-09150abf3d8b admin admin] PATH_INFO: `/v3/projects/87d9e193941c4afe8d81e422938ea996` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:05.939717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41a47e04-4078-4b06-931d-09150abf3d8b admin admin] RBAC: Authorizing `identity:delete_project(project_id=87d9e193941c4afe8d81e422938ea996)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:05.943619 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41a47e04-4078-4b06-931d-09150abf3d8b admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:05.951964 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3474bc-33dd-4dcc-a0b8-f7cddc2915ed None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:05.956476 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5052413e-d0aa-4235-9eec-3702768926df service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:05.957128 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5052413e-d0aa-4235-9eec-3702768926df service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:05.957415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5052413e-d0aa-4235-9eec-3702768926df service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:05.957697 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5052413e-d0aa-4235-9eec-3702768926df service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:05.959683 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-5052413e-d0aa-4235-9eec-3702768926df service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:06.022355 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-41a47e04-4078-4b06-931d-09150abf3d8b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '87d9e193941c4afe8d81e422938ea996', 'request_id': 'req-41a47e04-4078-4b06-931d-09150abf3d8b'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:06.032802 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-41a47e04-4078-4b06-931d-09150abf3d8b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '87d9e193941c4afe8d81e422938ea996', 'request_id': 'req-41a47e04-4078-4b06-931d-09150abf3d8b'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:06.035993 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:06.045561 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-41a47e04-4078-4b06-931d-09150abf3d8b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '87d9e193941c4afe8d81e422938ea996', 'request_id': 'req-41a47e04-4078-4b06-931d-09150abf3d8b'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:06.055735 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5052413e-d0aa-4235-9eec-3702768926df service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:06.058236 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5052413e-d0aa-4235-9eec-3702768926df service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:06.063905 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-41a47e04-4078-4b06-931d-09150abf3d8b admin admin] The token cache is being invalidate because project 87d9e193941c4afe8d81e422938ea996 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:53:06.064244 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-41a47e04-4078-4b06-931d-09150abf3d8b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:06.065762 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-41a47e04-4078-4b06-931d-09150abf3d8b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:06.067438 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1272/2559] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:53:05 2022] DELETE /identity/v3/projects/87d9e193941c4afe8d81e422938ea996 => generated 0 bytes in 290 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:06.068275 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41a47e04-4078-4b06-931d-09150abf3d8b admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:06.069780 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-41a47e04-4078-4b06-931d-09150abf3d8b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:06.073160 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1288/2560] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:53:05 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 193 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:53:06.074456 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5052413e-d0aa-4235-9eec-3702768926df service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:06.097222 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5052413e-d0aa-4235-9eec-3702768926df service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:06.174624 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41a47e04-4078-4b06-931d-09150abf3d8b admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:06.180553 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b78574-5064-400b-84c5-5ac86295da65 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:06.181580 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b78574-5064-400b-84c5-5ac86295da65 service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:06.182014 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b78574-5064-400b-84c5-5ac86295da65 service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:06.182559 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b78574-5064-400b-84c5-5ac86295da65 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:06.185094 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-76b78574-5064-400b-84c5-5ac86295da65 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:06.201953 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06aef99-f1c4-495d-b3d2-0bb645c4685c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:06.203019 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06aef99-f1c4-495d-b3d2-0bb645c4685c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:06.203492 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06aef99-f1c4-495d-b3d2-0bb645c4685c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:06.203959 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06aef99-f1c4-495d-b3d2-0bb645c4685c admin admin] PATH_INFO: `/v3/users/0fb9a0a552ae468dbc3935d515c63556` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:06.207115 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c06aef99-f1c4-495d-b3d2-0bb645c4685c admin admin] RBAC: Authorizing `identity:delete_user(user_id=0fb9a0a552ae468dbc3935d515c63556)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:06.210672 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c06aef99-f1c4-495d-b3d2-0bb645c4685c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:06.226179 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:06.308440 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c06aef99-f1c4-495d-b3d2-0bb645c4685c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0fb9a0a552ae468dbc3935d515c63556'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:06.325414 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c06aef99-f1c4-495d-b3d2-0bb645c4685c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0fb9a0a552ae468dbc3935d515c63556'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:06.325976 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76b78574-5064-400b-84c5-5ac86295da65 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:06.327390 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c06aef99-f1c4-495d-b3d2-0bb645c4685c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0fb9a0a552ae468dbc3935d515c63556'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:06.327724 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76b78574-5064-400b-84c5-5ac86295da65 service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:06.329870 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c06aef99-f1c4-495d-b3d2-0bb645c4685c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0fb9a0a552ae468dbc3935d515c63556'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:06.342594 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1273/2561] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:53:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 275 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:53:06.353944 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c06aef99-f1c4-495d-b3d2-0bb645c4685c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0fb9a0a552ae468dbc3935d515c63556'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:06.391297 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1289/2562] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:53:06 2022] DELETE /identity/v3/users/0fb9a0a552ae468dbc3935d515c63556 => generated 0 bytes in 318 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:07.144505 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b78574-5064-400b-84c5-5ac86295da65 service neutron] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:07.147407 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-76b78574-5064-400b-84c5-5ac86295da65 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:07.207766 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b78574-5064-400b-84c5-5ac86295da65 service neutron] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:07.308885 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f306e5f-4b39-4fc0-b4b6-2a001adda17c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:07.309664 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f306e5f-4b39-4fc0-b4b6-2a001adda17c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:07.310037 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f306e5f-4b39-4fc0-b4b6-2a001adda17c admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:07.311155 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f306e5f-4b39-4fc0-b4b6-2a001adda17c admin admin] PATH_INFO: `/v3/projects/d5934f936f52409f9fba8c7c28d3a32d` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:07.312497 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f306e5f-4b39-4fc0-b4b6-2a001adda17c admin admin] RBAC: Authorizing `identity:delete_project(project_id=d5934f936f52409f9fba8c7c28d3a32d)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:07.314502 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f306e5f-4b39-4fc0-b4b6-2a001adda17c admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:07.411030 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-0f306e5f-4b39-4fc0-b4b6-2a001adda17c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd5934f936f52409f9fba8c7c28d3a32d', 'request_id': 'req-0f306e5f-4b39-4fc0-b4b6-2a001adda17c'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:07.425065 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-0f306e5f-4b39-4fc0-b4b6-2a001adda17c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd5934f936f52409f9fba8c7c28d3a32d', 'request_id': 'req-0f306e5f-4b39-4fc0-b4b6-2a001adda17c'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:07.439885 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-0f306e5f-4b39-4fc0-b4b6-2a001adda17c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd5934f936f52409f9fba8c7c28d3a32d', 'request_id': 'req-0f306e5f-4b39-4fc0-b4b6-2a001adda17c'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:07.478731 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-0f306e5f-4b39-4fc0-b4b6-2a001adda17c admin admin] The token cache is being invalidate because project d5934f936f52409f9fba8c7c28d3a32d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:53:07.479075 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-0f306e5f-4b39-4fc0-b4b6-2a001adda17c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:07.479808 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-0f306e5f-4b39-4fc0-b4b6-2a001adda17c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:07.481651 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1274/2563] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:53:07 2022] DELETE /identity/v3/projects/d5934f936f52409f9fba8c7c28d3a32d => generated 0 bytes in 338 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:07.489452 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d35e0a-a5d6-4ba8-b6c1-32e10394cf5d None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:07.491830 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b512a7ae-48e3-4f55-8116-f0facdc3334e None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:07.492337 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b512a7ae-48e3-4f55-8116-f0facdc3334e None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:07.492662 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d35e0a-a5d6-4ba8-b6c1-32e10394cf5d None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:07.492822 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b512a7ae-48e3-4f55-8116-f0facdc3334e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:07.493131 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d35e0a-a5d6-4ba8-b6c1-32e10394cf5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:07.527258 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-66d35e0a-a5d6-4ba8-b6c1-32e10394cf5d None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:07.529534 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-66d35e0a-a5d6-4ba8-b6c1-32e10394cf5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:07.558724 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-b512a7ae-48e3-4f55-8116-f0facdc3334e None None] MFA Rules not processed for user `03af8e89b8434a038fe593252633de19`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:07.562774 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-b512a7ae-48e3-4f55-8116-f0facdc3334e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:07.637055 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:07.666038 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1290/2564] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:53:07.668319 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef3c0f2-9d26-4160-90b1-f2e09a8ece3c None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:07.668614 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef3c0f2-9d26-4160-90b1-f2e09a8ece3c None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:07.668963 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef3c0f2-9d26-4160-90b1-f2e09a8ece3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:07.694113 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-7ef3c0f2-9d26-4160-90b1-f2e09a8ece3c None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:07.696282 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7ef3c0f2-9d26-4160-90b1-f2e09a8ece3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:07.709381 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1291/2565] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:53:07.715082 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef3c0f2-9d26-4160-90b1-f2e09a8ece3c None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:07.723334 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1275/2566] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:07 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 233 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:53:07.727773 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:07.729149 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b512a7ae-48e3-4f55-8116-f0facdc3334e None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:07.735625 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-b512a7ae-48e3-4f55-8116-f0facdc3334e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:07.737429 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef3c0f2-9d26-4160-90b1-f2e09a8ece3c None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:07.745146 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09fd0c77-4b7d-43f2-a8a5-ee7ad31bad20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:07.747228 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09fd0c77-4b7d-43f2-a8a5-ee7ad31bad20 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:07.747661 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09fd0c77-4b7d-43f2-a8a5-ee7ad31bad20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:07.748265 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09fd0c77-4b7d-43f2-a8a5-ee7ad31bad20 admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:07.749058 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09fd0c77-4b7d-43f2-a8a5-ee7ad31bad20 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:53:07.749472 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09fd0c77-4b7d-43f2-a8a5-ee7ad31bad20 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:07.751550 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09fd0c77-4b7d-43f2-a8a5-ee7ad31bad20 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:07.773504 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1292/2567] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:53:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:53:07.775115 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09fd0c77-4b7d-43f2-a8a5-ee7ad31bad20 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:07.777044 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b512a7ae-48e3-4f55-8116-f0facdc3334e None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:07.788944 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09fd0c77-4b7d-43f2-a8a5-ee7ad31bad20 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:07.793139 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd325ba-a6a3-4951-9a01-a44d4743c505 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:07.794214 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd325ba-a6a3-4951-9a01-a44d4743c505 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:07.794933 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd325ba-a6a3-4951-9a01-a44d4743c505 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:07.795411 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd325ba-a6a3-4951-9a01-a44d4743c505 admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:07.796525 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fd325ba-a6a3-4951-9a01-a44d4743c505 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:07.798718 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc235e45-0b14-4c6d-99d8-b6035fe8c043 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:07.799599 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc235e45-0b14-4c6d-99d8-b6035fe8c043 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:07.800041 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc235e45-0b14-4c6d-99d8-b6035fe8c043 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:07.800473 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc235e45-0b14-4c6d-99d8-b6035fe8c043 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:07.801023 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fd325ba-a6a3-4951-9a01-a44d4743c505 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:07.801582 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc235e45-0b14-4c6d-99d8-b6035fe8c043 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:07.803751 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc235e45-0b14-4c6d-99d8-b6035fe8c043 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:07.851631 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1293/2568] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:53:07 2022] POST /identity/v3/projects => generated 367 bytes in 78 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:53:07.853129 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc235e45-0b14-4c6d-99d8-b6035fe8c043 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:07.882152 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1276/2569] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:53:07 2022] POST /identity/v3/projects => generated 371 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:53:07.890704 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd325ba-a6a3-4951-9a01-a44d4743c505 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:07.982315 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc235e45-0b14-4c6d-99d8-b6035fe8c043 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:08.026668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd325ba-a6a3-4951-9a01-a44d4743c505 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:08.034945 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7fa5b7-078e-4fee-aeb6-ec81dba00a59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:08.036010 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7fa5b7-078e-4fee-aeb6-ec81dba00a59 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:08.036493 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7fa5b7-078e-4fee-aeb6-ec81dba00a59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:08.036965 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7fa5b7-078e-4fee-aeb6-ec81dba00a59 admin admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:08.038132 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7fa5b7-078e-4fee-aeb6-ec81dba00a59 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:08.041974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7fa5b7-078e-4fee-aeb6-ec81dba00a59 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:08.082377 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1277/2570] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:53:07 2022] POST /identity/v3/users => generated 292 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:53:08.083767 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1cd5b33-c4c6-4cb3-950a-43f1d877da9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:08.084931 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cd5b33-c4c6-4cb3-950a-43f1d877da9e admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:08.085504 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cd5b33-c4c6-4cb3-950a-43f1d877da9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:08.086080 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1cd5b33-c4c6-4cb3-950a-43f1d877da9e admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:08.087543 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1cd5b33-c4c6-4cb3-950a-43f1d877da9e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:08.088167 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7fa5b7-078e-4fee-aeb6-ec81dba00a59 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:08.091365 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1cd5b33-c4c6-4cb3-950a-43f1d877da9e admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:08.106813 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7fa5b7-078e-4fee-aeb6-ec81dba00a59 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:08.117160 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4209d79b-27f3-4a7d-b189-e00af9a60719 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:08.119036 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4209d79b-27f3-4a7d-b189-e00af9a60719 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:08.119859 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4209d79b-27f3-4a7d-b189-e00af9a60719 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:08.120346 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4209d79b-27f3-4a7d-b189-e00af9a60719 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:08.121550 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4209d79b-27f3-4a7d-b189-e00af9a60719 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:08.125467 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4209d79b-27f3-4a7d-b189-e00af9a60719 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:08.129333 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1294/2571] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:53:07 2022] POST /identity/v3/users => generated 284 bytes in 277 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:53:08.135382 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1cd5b33-c4c6-4cb3-950a-43f1d877da9e admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:08.150525 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1cd5b33-c4c6-4cb3-950a-43f1d877da9e admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:08.156468 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1278/2572] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:53:08 2022] GET /identity/v3/roles => generated 1420 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:53:08.157220 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4564cc0-e800-4661-a14f-87b16401a863 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:08.157787 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4564cc0-e800-4661-a14f-87b16401a863 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:08.158075 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4564cc0-e800-4661-a14f-87b16401a863 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:08.158219 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4209d79b-27f3-4a7d-b189-e00af9a60719 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:08.158587 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4564cc0-e800-4661-a14f-87b16401a863 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:08.159048 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4564cc0-e800-4661-a14f-87b16401a863 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:08.160773 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4564cc0-e800-4661-a14f-87b16401a863 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:08.174552 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1295/2573] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:53:08 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:53:08.180067 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4564cc0-e800-4661-a14f-87b16401a863 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:08.183632 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4209d79b-27f3-4a7d-b189-e00af9a60719 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:08.197674 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c7d54e-2fe2-4631-85ed-39c9c0f36203 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:08.198897 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c7d54e-2fe2-4631-85ed-39c9c0f36203 admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:08.199097 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4564cc0-e800-4661-a14f-87b16401a863 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:08.199399 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c7d54e-2fe2-4631-85ed-39c9c0f36203 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:08.199981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c7d54e-2fe2-4631-85ed-39c9c0f36203 admin admin] PATH_INFO: `/v3/projects/44b475d014c44b189041dda9b761576e/users/39a6689ef920461982651b13d4cbc65c/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:08.210573 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0cdd8a-a192-45fb-9051-c4fbf499235f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:08.211770 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0cdd8a-a192-45fb-9051-c4fbf499235f admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:08.212322 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0cdd8a-a192-45fb-9051-c4fbf499235f admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:08.212883 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0cdd8a-a192-45fb-9051-c4fbf499235f admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:08.214176 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a0cdd8a-a192-45fb-9051-c4fbf499235f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:08.217984 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a0cdd8a-a192-45fb-9051-c4fbf499235f admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:08.241375 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c7d54e-2fe2-4631-85ed-39c9c0f36203 admin admin] RBAC: Authorizing `identity:create_grant(project_id=44b475d014c44b189041dda9b761576e, user_id=39a6689ef920461982651b13d4cbc65c, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:08.245268 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1296/2574] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:53:08 2022] GET /identity/v3/roles => generated 1420 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:53:08.252051 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c7d54e-2fe2-4631-85ed-39c9c0f36203 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:08.252774 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0cdd8a-a192-45fb-9051-c4fbf499235f admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:08.270142 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0cdd8a-a192-45fb-9051-c4fbf499235f admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:08.280418 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003db236-875a-4001-b542-3cd85c0b3890 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:08.282101 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003db236-875a-4001-b542-3cd85c0b3890 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:08.282688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003db236-875a-4001-b542-3cd85c0b3890 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:08.283268 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003db236-875a-4001-b542-3cd85c0b3890 admin admin] PATH_INFO: `/v3/projects/90c775ca8d20433b94a04374ff7813f2/users/64dbc56c09f84f11912c51f3515d484b/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:08.292961 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1279/2575] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:53:08 2022] PUT /identity/v3/projects/44b475d014c44b189041dda9b761576e/users/39a6689ef920461982651b13d4cbc65c/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:08.300151 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c7d54e-2fe2-4631-85ed-39c9c0f36203 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:08.311702 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-003db236-875a-4001-b542-3cd85c0b3890 admin admin] RBAC: Authorizing `identity:create_grant(project_id=90c775ca8d20433b94a04374ff7813f2, user_id=64dbc56c09f84f11912c51f3515d484b, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:08.314712 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-003db236-875a-4001-b542-3cd85c0b3890 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:08.337627 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1297/2576] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:53:08 2022] PUT /identity/v3/projects/90c775ca8d20433b94a04374ff7813f2/users/64dbc56c09f84f11912c51f3515d484b/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 86 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:08.349930 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01978e8b-3db5-482d-ba9a-b587a34c4b53 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:08.350257 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01978e8b-3db5-482d-ba9a-b587a34c4b53 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:08.350614 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01978e8b-3db5-482d-ba9a-b587a34c4b53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:08.387290 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-01978e8b-3db5-482d-ba9a-b587a34c4b53 None None] MFA Rules not processed for user `64dbc56c09f84f11912c51f3515d484b`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:08.388089 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-01978e8b-3db5-482d-ba9a-b587a34c4b53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:08.395364 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:08.453919 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c7d54e-2fe2-4631-85ed-39c9c0f36203 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:08.470104 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:08.515908 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1298/2577] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:08 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:53:08.520432 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01978e8b-3db5-482d-ba9a-b587a34c4b53 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:08.563924 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925dec05-e31f-45ff-a8c6-62c439eeac3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:08.565157 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925dec05-e31f-45ff-a8c6-62c439eeac3d admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:08.565808 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925dec05-e31f-45ff-a8c6-62c439eeac3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:08.566282 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925dec05-e31f-45ff-a8c6-62c439eeac3d admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:08.567236 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-925dec05-e31f-45ff-a8c6-62c439eeac3d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:08.569147 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-925dec05-e31f-45ff-a8c6-62c439eeac3d admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:08.586174 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1280/2578] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:53:08 2022] GET /identity/v3/roles => generated 1420 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:53:08.587405 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925dec05-e31f-45ff-a8c6-62c439eeac3d admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:08.602552 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925dec05-e31f-45ff-a8c6-62c439eeac3d admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:08.615405 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800c0bb6-dafd-413a-a838-f3e58e4fd3f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:08.616339 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800c0bb6-dafd-413a-a838-f3e58e4fd3f7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:08.617342 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800c0bb6-dafd-413a-a838-f3e58e4fd3f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:08.617646 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800c0bb6-dafd-413a-a838-f3e58e4fd3f7 admin admin] PATH_INFO: `/v3/projects/44b475d014c44b189041dda9b761576e/users/39a6689ef920461982651b13d4cbc65c/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:08.620698 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-800c0bb6-dafd-413a-a838-f3e58e4fd3f7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=44b475d014c44b189041dda9b761576e, user_id=39a6689ef920461982651b13d4cbc65c, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:08.624439 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-800c0bb6-dafd-413a-a838-f3e58e4fd3f7 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:08.638203 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1281/2579] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:53:08 2022] PUT /identity/v3/projects/44b475d014c44b189041dda9b761576e/users/39a6689ef920461982651b13d4cbc65c/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:08.657523 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01978e8b-3db5-482d-ba9a-b587a34c4b53 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:08.775126 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae68a30c-8361-4b37-bb0c-aa0f230c6ebc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:08.776177 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae68a30c-8361-4b37-bb0c-aa0f230c6ebc admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:08.776722 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae68a30c-8361-4b37-bb0c-aa0f230c6ebc admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:08.777220 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae68a30c-8361-4b37-bb0c-aa0f230c6ebc admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:08.778437 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae68a30c-8361-4b37-bb0c-aa0f230c6ebc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:08.781691 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae68a30c-8361-4b37-bb0c-aa0f230c6ebc admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:08.837173 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1299/2580] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:53:08 2022] POST /identity/v3/projects => generated 367 bytes in 314 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:53:08.838664 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800c0bb6-dafd-413a-a838-f3e58e4fd3f7 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:08.852877 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800c0bb6-dafd-413a-a838-f3e58e4fd3f7 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:08.949244 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c0f3c8-71e6-40f7-bea6-ff9ce49535f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:08.950158 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c0f3c8-71e6-40f7-bea6-ff9ce49535f9 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:08.950754 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c0f3c8-71e6-40f7-bea6-ff9ce49535f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:08.951239 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c0f3c8-71e6-40f7-bea6-ff9ce49535f9 admin admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:08.952075 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54c0f3c8-71e6-40f7-bea6-ff9ce49535f9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:08.954596 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54c0f3c8-71e6-40f7-bea6-ff9ce49535f9 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:08.994336 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1282/2581] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:53:08 2022] POST /identity/v3/users => generated 290 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:53:08.996354 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae68a30c-8361-4b37-bb0c-aa0f230c6ebc admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:09.016801 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae68a30c-8361-4b37-bb0c-aa0f230c6ebc admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:09.025591 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4370f151-a4fa-4d70-86a4-722fe0e6d1a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:09.026262 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4370f151-a4fa-4d70-86a4-722fe0e6d1a1 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:09.026626 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4370f151-a4fa-4d70-86a4-722fe0e6d1a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:09.027014 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4370f151-a4fa-4d70-86a4-722fe0e6d1a1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:09.028353 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4370f151-a4fa-4d70-86a4-722fe0e6d1a1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:09.031724 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4370f151-a4fa-4d70-86a4-722fe0e6d1a1 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:09.052553 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1300/2582] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:53:08 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:53:09.057016 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c0f3c8-71e6-40f7-bea6-ff9ce49535f9 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:09.069162 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c0f3c8-71e6-40f7-bea6-ff9ce49535f9 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:09.074460 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184f5d77-1038-481f-b174-9c3cf4148efd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:09.075168 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184f5d77-1038-481f-b174-9c3cf4148efd admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:09.076325 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184f5d77-1038-481f-b174-9c3cf4148efd admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:09.076325 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184f5d77-1038-481f-b174-9c3cf4148efd admin admin] PATH_INFO: `/v3/projects/9492825663f04194b6989ef7c64cbf82/users/279f947306ef4b289c94e26cdfdac74b/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:09.095310 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-184f5d77-1038-481f-b174-9c3cf4148efd admin admin] RBAC: Authorizing `identity:create_grant(project_id=9492825663f04194b6989ef7c64cbf82, user_id=279f947306ef4b289c94e26cdfdac74b, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:09.098895 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-184f5d77-1038-481f-b174-9c3cf4148efd admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:09.136484 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1283/2583] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:53:09 2022] PUT /identity/v3/projects/9492825663f04194b6989ef7c64cbf82/users/279f947306ef4b289c94e26cdfdac74b/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 80 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:53:09.141885 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4370f151-a4fa-4d70-86a4-722fe0e6d1a1 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:09.160043 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4370f151-a4fa-4d70-86a4-722fe0e6d1a1 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:09.215259 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:09.286338 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de12c57-86b0-405d-9539-9ea8f7cc8d3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:09.287511 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de12c57-86b0-405d-9539-9ea8f7cc8d3e admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:09.288203 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de12c57-86b0-405d-9539-9ea8f7cc8d3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:09.288709 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de12c57-86b0-405d-9539-9ea8f7cc8d3e admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:09.289730 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3de12c57-86b0-405d-9539-9ea8f7cc8d3e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:09.291725 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3de12c57-86b0-405d-9539-9ea8f7cc8d3e admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:09.312989 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184f5d77-1038-481f-b174-9c3cf4148efd admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:09.314243 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1301/2584] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:53:09 2022] GET /identity/v3/roles => generated 1420 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:53:09.331890 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184f5d77-1038-481f-b174-9c3cf4148efd admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:09.338797 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10add140-a940-47c8-911f-f870195140e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:09.339946 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10add140-a940-47c8-911f-f870195140e0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:09.340652 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10add140-a940-47c8-911f-f870195140e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:09.341166 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10add140-a940-47c8-911f-f870195140e0 admin admin] PATH_INFO: `/v3/projects/9492825663f04194b6989ef7c64cbf82/users/279f947306ef4b289c94e26cdfdac74b/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:09.344237 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10add140-a940-47c8-911f-f870195140e0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9492825663f04194b6989ef7c64cbf82, user_id=279f947306ef4b289c94e26cdfdac74b, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:09.347715 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10add140-a940-47c8-911f-f870195140e0 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:09.374009 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ffed23-8320-421c-901c-c3332e1f7821 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:09.374757 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ffed23-8320-421c-901c-c3332e1f7821 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:09.375252 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ffed23-8320-421c-901c-c3332e1f7821 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:09.379197 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1284/2585] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:53:09 2022] PUT /identity/v3/projects/9492825663f04194b6989ef7c64cbf82/users/279f947306ef4b289c94e26cdfdac74b/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:09.414058 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-27ffed23-8320-421c-901c-c3332e1f7821 None None] MFA Rules not processed for user `279f947306ef4b289c94e26cdfdac74b`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:09.415829 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-27ffed23-8320-421c-901c-c3332e1f7821 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:09.503421 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1302/2586] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:09 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:53:09.517440 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10add140-a940-47c8-911f-f870195140e0 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:09.519646 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-10add140-a940-47c8-911f-f870195140e0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:09.614538 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:09.628033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10add140-a940-47c8-911f-f870195140e0 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:09.636441 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d2055e-82c7-4dde-bf83-64359354e079 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:09.637413 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d2055e-82c7-4dde-bf83-64359354e079 service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:09.637847 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d2055e-82c7-4dde-bf83-64359354e079 service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:09.646705 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73d2055e-82c7-4dde-bf83-64359354e079 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:09.662350 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73d2055e-82c7-4dde-bf83-64359354e079 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:09.664799 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73d2055e-82c7-4dde-bf83-64359354e079 service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:09.681790 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1285/2587] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:53:09 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 165 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:53:09.773484 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ae18a7-3aa6-47e8-ad40-8c26ed7bd8d9 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:09.774318 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ae18a7-3aa6-47e8-ad40-8c26ed7bd8d9 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:09.774318 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ae18a7-3aa6-47e8-ad40-8c26ed7bd8d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:09.817896 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-45ae18a7-3aa6-47e8-ad40-8c26ed7bd8d9 None None] MFA Rules not processed for user `39a6689ef920461982651b13d4cbc65c`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:09.823280 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-45ae18a7-3aa6-47e8-ad40-8c26ed7bd8d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:09.959488 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1303/2588] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:09 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:53:09.973612 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d2055e-82c7-4dde-bf83-64359354e079 service cinder] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:10.002520 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73d2055e-82c7-4dde-bf83-64359354e079 service cinder] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:10.015357 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d11a42a-cee5-4578-9d3b-d4124e3eee6c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:10.016463 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d11a42a-cee5-4578-9d3b-d4124e3eee6c service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:10.017007 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d11a42a-cee5-4578-9d3b-d4124e3eee6c service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:10.017488 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d11a42a-cee5-4578-9d3b-d4124e3eee6c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:10.154894 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d11a42a-cee5-4578-9d3b-d4124e3eee6c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:10.157750 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d11a42a-cee5-4578-9d3b-d4124e3eee6c service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:10.188771 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1286/2589] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:53:09 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 216 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:53:11.803784 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ae18a7-3aa6-47e8-ad40-8c26ed7bd8d9 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:11.857701 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=100 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:11.889530 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ae18a7-3aa6-47e8-ad40-8c26ed7bd8d9 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:11.894624 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d11a42a-cee5-4578-9d3b-d4124e3eee6c service cinder] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:11.895865 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7077e787-ecd9-449f-acf1-e579a95c8ddb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:11.896957 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7077e787-ecd9-449f-acf1-e579a95c8ddb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:11.897441 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7077e787-ecd9-449f-acf1-e579a95c8ddb admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:11.897905 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7077e787-ecd9-449f-acf1-e579a95c8ddb admin admin] PATH_INFO: `/v3/users/64dbc56c09f84f11912c51f3515d484b` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:11.900591 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7077e787-ecd9-449f-acf1-e579a95c8ddb admin admin] RBAC: Authorizing `identity:delete_user(user_id=64dbc56c09f84f11912c51f3515d484b)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:11.904428 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7077e787-ecd9-449f-acf1-e579a95c8ddb admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:11.905689 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d11a42a-cee5-4578-9d3b-d4124e3eee6c service cinder] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:11.910783 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185719eb-db8a-4531-bf80-cd65a2fad188 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:11.911409 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185719eb-db8a-4531-bf80-cd65a2fad188 service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:11.911688 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185719eb-db8a-4531-bf80-cd65a2fad188 service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:11.911960 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185719eb-db8a-4531-bf80-cd65a2fad188 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:11.923068 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-185719eb-db8a-4531-bf80-cd65a2fad188 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:11.924950 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-185719eb-db8a-4531-bf80-cd65a2fad188 service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:11.937507 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1287/2590] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:53:11 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:53:11.976248 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7077e787-ecd9-449f-acf1-e579a95c8ddb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '64dbc56c09f84f11912c51f3515d484b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:11.986553 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7077e787-ecd9-449f-acf1-e579a95c8ddb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '64dbc56c09f84f11912c51f3515d484b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:11.987226 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7077e787-ecd9-449f-acf1-e579a95c8ddb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '64dbc56c09f84f11912c51f3515d484b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:11.987859 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7077e787-ecd9-449f-acf1-e579a95c8ddb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '64dbc56c09f84f11912c51f3515d484b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:11.997127 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7077e787-ecd9-449f-acf1-e579a95c8ddb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '64dbc56c09f84f11912c51f3515d484b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:12.031003 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1304/2591] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:53:11 2022] DELETE /identity/v3/users/64dbc56c09f84f11912c51f3515d484b => generated 0 bytes in 228 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:12.036895 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6286efb1-230e-4376-b1e0-17d036fa2816 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:12.037358 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6286efb1-230e-4376-b1e0-17d036fa2816 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:12.037772 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6286efb1-230e-4376-b1e0-17d036fa2816 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:12.064928 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-6286efb1-230e-4376-b1e0-17d036fa2816 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:12.066193 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-6286efb1-230e-4376-b1e0-17d036fa2816 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:12.167305 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1288/2592] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:53:12.180473 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7077e787-ecd9-449f-acf1-e579a95c8ddb admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:12.182157 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7077e787-ecd9-449f-acf1-e579a95c8ddb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:12.286861 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7077e787-ecd9-449f-acf1-e579a95c8ddb admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:12.293698 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05994bc2-12ea-4511-8fb6-e42e3912da45 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:12.295300 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05994bc2-12ea-4511-8fb6-e42e3912da45 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:12.295777 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05994bc2-12ea-4511-8fb6-e42e3912da45 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:12.296252 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05994bc2-12ea-4511-8fb6-e42e3912da45 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:12.308329 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05994bc2-12ea-4511-8fb6-e42e3912da45 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:12.309990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05994bc2-12ea-4511-8fb6-e42e3912da45 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:12.323650 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1305/2593] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:53:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:53:12.702582 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6286efb1-230e-4376-b1e0-17d036fa2816 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:12.705081 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-6286efb1-230e-4376-b1e0-17d036fa2816 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:12.724271 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6286efb1-230e-4376-b1e0-17d036fa2816 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:12.730172 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f39cab-0b15-4871-b599-7d7ef4744629 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:12.730926 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f39cab-0b15-4871-b599-7d7ef4744629 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:12.731271 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f39cab-0b15-4871-b599-7d7ef4744629 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:12.731611 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f39cab-0b15-4871-b599-7d7ef4744629 admin admin] PATH_INFO: `/v3/projects/90c775ca8d20433b94a04374ff7813f2` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:12.733945 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1f39cab-0b15-4871-b599-7d7ef4744629 admin admin] RBAC: Authorizing `identity:delete_project(project_id=90c775ca8d20433b94a04374ff7813f2)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:12.738676 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1f39cab-0b15-4871-b599-7d7ef4744629 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:12.824161 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-b1f39cab-0b15-4871-b599-7d7ef4744629 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '90c775ca8d20433b94a04374ff7813f2', 'request_id': 'req-b1f39cab-0b15-4871-b599-7d7ef4744629'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:12.837408 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-b1f39cab-0b15-4871-b599-7d7ef4744629 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '90c775ca8d20433b94a04374ff7813f2', 'request_id': 'req-b1f39cab-0b15-4871-b599-7d7ef4744629'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:12.850314 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-b1f39cab-0b15-4871-b599-7d7ef4744629 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '90c775ca8d20433b94a04374ff7813f2', 'request_id': 'req-b1f39cab-0b15-4871-b599-7d7ef4744629'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:12.863522 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:12.872897 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-b1f39cab-0b15-4871-b599-7d7ef4744629 admin admin] The token cache is being invalidate because project 90c775ca8d20433b94a04374ff7813f2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:53:12.873486 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-b1f39cab-0b15-4871-b599-7d7ef4744629 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:12.875394 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-b1f39cab-0b15-4871-b599-7d7ef4744629 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:12.878710 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1289/2594] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:53:12 2022] DELETE /identity/v3/projects/90c775ca8d20433b94a04374ff7813f2 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:12.884746 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05994bc2-12ea-4511-8fb6-e42e3912da45 service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:12.904519 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05994bc2-12ea-4511-8fb6-e42e3912da45 service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:12.990046 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:12.997842 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e232e209-43d9-4f1f-9585-ea1758780a3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:12.998628 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e232e209-43d9-4f1f-9585-ea1758780a3d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:12.999527 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e232e209-43d9-4f1f-9585-ea1758780a3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:12.999893 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e232e209-43d9-4f1f-9585-ea1758780a3d admin admin] PATH_INFO: `/v3/users/279f947306ef4b289c94e26cdfdac74b` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:13.002223 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e232e209-43d9-4f1f-9585-ea1758780a3d admin admin] RBAC: Authorizing `identity:delete_user(user_id=279f947306ef4b289c94e26cdfdac74b)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:13.005585 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e232e209-43d9-4f1f-9585-ea1758780a3d admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:13.066020 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e232e209-43d9-4f1f-9585-ea1758780a3d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '279f947306ef4b289c94e26cdfdac74b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:13.075558 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e232e209-43d9-4f1f-9585-ea1758780a3d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '279f947306ef4b289c94e26cdfdac74b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:13.076811 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e232e209-43d9-4f1f-9585-ea1758780a3d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '279f947306ef4b289c94e26cdfdac74b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:13.077529 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e232e209-43d9-4f1f-9585-ea1758780a3d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '279f947306ef4b289c94e26cdfdac74b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:13.086757 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e232e209-43d9-4f1f-9585-ea1758780a3d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '279f947306ef4b289c94e26cdfdac74b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:13.113096 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1306/2595] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:53:12 2022] DELETE /identity/v3/users/279f947306ef4b289c94e26cdfdac74b => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:13.619393 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f39cab-0b15-4871-b599-7d7ef4744629 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:13.621785 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-b1f39cab-0b15-4871-b599-7d7ef4744629 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:13.639966 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f39cab-0b15-4871-b599-7d7ef4744629 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:13.751356 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a5ac9c-6003-4e1b-a75e-ef2a83d06495 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:13.752108 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a5ac9c-6003-4e1b-a75e-ef2a83d06495 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:13.752485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a5ac9c-6003-4e1b-a75e-ef2a83d06495 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:13.752838 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a5ac9c-6003-4e1b-a75e-ef2a83d06495 admin admin] PATH_INFO: `/v3/projects/9492825663f04194b6989ef7c64cbf82` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:13.754175 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a5ac9c-6003-4e1b-a75e-ef2a83d06495 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9492825663f04194b6989ef7c64cbf82)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:13.756326 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a5ac9c-6003-4e1b-a75e-ef2a83d06495 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:13.839897 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-33a5ac9c-6003-4e1b-a75e-ef2a83d06495 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9492825663f04194b6989ef7c64cbf82', 'request_id': 'req-33a5ac9c-6003-4e1b-a75e-ef2a83d06495'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:13.859872 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-33a5ac9c-6003-4e1b-a75e-ef2a83d06495 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9492825663f04194b6989ef7c64cbf82', 'request_id': 'req-33a5ac9c-6003-4e1b-a75e-ef2a83d06495'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:13.873863 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-33a5ac9c-6003-4e1b-a75e-ef2a83d06495 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9492825663f04194b6989ef7c64cbf82', 'request_id': 'req-33a5ac9c-6003-4e1b-a75e-ef2a83d06495'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:13.890901 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-33a5ac9c-6003-4e1b-a75e-ef2a83d06495 admin admin] The token cache is being invalidate because project 9492825663f04194b6989ef7c64cbf82 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:53:13.891129 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-33a5ac9c-6003-4e1b-a75e-ef2a83d06495 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:13.892139 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-33a5ac9c-6003-4e1b-a75e-ef2a83d06495 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:13.894346 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1290/2596] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:53:13 2022] DELETE /identity/v3/projects/9492825663f04194b6989ef7c64cbf82 => generated 0 bytes in 276 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:53:13.899447 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c279b17-13b2-4a54-8128-75b009a2d9c1 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:13.900431 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c279b17-13b2-4a54-8128-75b009a2d9c1 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:13.900642 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c279b17-13b2-4a54-8128-75b009a2d9c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:13.927593 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-9c279b17-13b2-4a54-8128-75b009a2d9c1 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:13.929858 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-9c279b17-13b2-4a54-8128-75b009a2d9c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:14.019176 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1307/2597] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:53:14.028920 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee20a43-946d-4312-ab42-853732ad201c None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:14.029396 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee20a43-946d-4312-ab42-853732ad201c None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:14.029879 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ee20a43-946d-4312-ab42-853732ad201c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:14.064547 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-8ee20a43-946d-4312-ab42-853732ad201c None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:14.065885 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-8ee20a43-946d-4312-ab42-853732ad201c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:14.076735 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1291/2598] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:53:14.080978 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c279b17-13b2-4a54-8128-75b009a2d9c1 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:14.098189 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c279b17-13b2-4a54-8128-75b009a2d9c1 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:14.107325 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3739103a-b76e-4f2f-a748-bc11a5303a02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:14.108555 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3739103a-b76e-4f2f-a748-bc11a5303a02 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:14.109234 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3739103a-b76e-4f2f-a748-bc11a5303a02 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:14.110001 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3739103a-b76e-4f2f-a748-bc11a5303a02 admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:14.111218 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3739103a-b76e-4f2f-a748-bc11a5303a02 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:53:14.111974 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3739103a-b76e-4f2f-a748-bc11a5303a02 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:14.114742 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3739103a-b76e-4f2f-a748-bc11a5303a02 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:14.139074 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1308/2599] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:53:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:53:14.143983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee20a43-946d-4312-ab42-853732ad201c None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:14.155626 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ee20a43-946d-4312-ab42-853732ad201c None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:14.161938 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb7348e-b184-4b67-984d-9c297e42c453 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:14.162760 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb7348e-b184-4b67-984d-9c297e42c453 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:14.163113 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb7348e-b184-4b67-984d-9c297e42c453 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:14.163452 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb7348e-b184-4b67-984d-9c297e42c453 admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:14.164195 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eb7348e-b184-4b67-984d-9c297e42c453 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:14.166678 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eb7348e-b184-4b67-984d-9c297e42c453 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:14.211919 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1292/2600] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:53:14 2022] POST /identity/v3/projects => generated 379 bytes in 69 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:53:14.216061 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3739103a-b76e-4f2f-a748-bc11a5303a02 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:14.230737 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3739103a-b76e-4f2f-a748-bc11a5303a02 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:14.324197 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5144deb4-50e4-45a7-9d55-aa6691a561e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:14.325159 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5144deb4-50e4-45a7-9d55-aa6691a561e2 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:14.325695 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5144deb4-50e4-45a7-9d55-aa6691a561e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:14.326800 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5144deb4-50e4-45a7-9d55-aa6691a561e2 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:14.327994 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5144deb4-50e4-45a7-9d55-aa6691a561e2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:14.331937 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5144deb4-50e4-45a7-9d55-aa6691a561e2 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:14.348662 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=87,keystone:DELETE=10,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:14.369110 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1309/2601] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:53:14 2022] POST /identity/v3/users => generated 290 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:53:14.375379 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb7348e-b184-4b67-984d-9c297e42c453 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:14.386190 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb7348e-b184-4b67-984d-9c297e42c453 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:14.391066 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7808b0da-bf0e-44e9-92a0-2909a18d9523 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:14.391680 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7808b0da-bf0e-44e9-92a0-2909a18d9523 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:14.391960 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7808b0da-bf0e-44e9-92a0-2909a18d9523 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:14.392225 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7808b0da-bf0e-44e9-92a0-2909a18d9523 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:14.392995 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7808b0da-bf0e-44e9-92a0-2909a18d9523 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:14.394580 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7808b0da-bf0e-44e9-92a0-2909a18d9523 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:14.406678 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1293/2602] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:53:14 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:53:14.411454 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5144deb4-50e4-45a7-9d55-aa6691a561e2 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:14.423890 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5144deb4-50e4-45a7-9d55-aa6691a561e2 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:14.430195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90932893-e9cb-4ab6-b24f-78f7ce77fc9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:14.431284 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90932893-e9cb-4ab6-b24f-78f7ce77fc9b admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:14.431810 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90932893-e9cb-4ab6-b24f-78f7ce77fc9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:14.432276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90932893-e9cb-4ab6-b24f-78f7ce77fc9b admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:14.433455 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90932893-e9cb-4ab6-b24f-78f7ce77fc9b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:14.436934 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90932893-e9cb-4ab6-b24f-78f7ce77fc9b admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:14.457979 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1310/2603] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:53:14 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:53:14.462788 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7808b0da-bf0e-44e9-92a0-2909a18d9523 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:14.473800 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7808b0da-bf0e-44e9-92a0-2909a18d9523 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:14.478754 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cbf5608-0167-4992-9272-af8cec815432 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:14.479441 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbf5608-0167-4992-9272-af8cec815432 admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:14.479756 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbf5608-0167-4992-9272-af8cec815432 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:14.480033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbf5608-0167-4992-9272-af8cec815432 admin admin] PATH_INFO: `/v3/projects/16939c0c855a44cba5d357baf68bc9be/users/a8e351115d2b449c9544905c2db91e69/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:14.497128 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cbf5608-0167-4992-9272-af8cec815432 admin admin] RBAC: Authorizing `identity:create_grant(project_id=16939c0c855a44cba5d357baf68bc9be, user_id=a8e351115d2b449c9544905c2db91e69, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:14.500378 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cbf5608-0167-4992-9272-af8cec815432 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:14.522896 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1294/2604] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:53:14 2022] PUT /identity/v3/projects/16939c0c855a44cba5d357baf68bc9be/users/a8e351115d2b449c9544905c2db91e69/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:14.535792 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd843fb-f82d-4b8c-8557-8b73601e2552 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:14.536118 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd843fb-f82d-4b8c-8557-8b73601e2552 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:14.536448 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fd843fb-f82d-4b8c-8557-8b73601e2552 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:14.571360 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-4fd843fb-f82d-4b8c-8557-8b73601e2552 None None] MFA Rules not processed for user `a8e351115d2b449c9544905c2db91e69`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:14.572630 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4fd843fb-f82d-4b8c-8557-8b73601e2552 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:14.668470 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1311/2605] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:14 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:53:14.673786 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cbf5608-0167-4992-9272-af8cec815432 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:14.710446 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:14.753104 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cbf5608-0167-4992-9272-af8cec815432 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:14.757965 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-277b83d5-25a7-40c8-b719-6ec3e5396802 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:14.758637 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277b83d5-25a7-40c8-b719-6ec3e5396802 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:14.758921 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277b83d5-25a7-40c8-b719-6ec3e5396802 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:14.759197 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277b83d5-25a7-40c8-b719-6ec3e5396802 admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:14.759852 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-277b83d5-25a7-40c8-b719-6ec3e5396802 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:14.761811 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-277b83d5-25a7-40c8-b719-6ec3e5396802 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:14.792910 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1295/2606] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:53:14 2022] POST /identity/v3/projects => generated 379 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:53:14.799261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd843fb-f82d-4b8c-8557-8b73601e2552 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:14.811659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fd843fb-f82d-4b8c-8557-8b73601e2552 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:14.886250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1640636-b1e8-4b58-bdf8-69b7959c00b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:14.886594 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1640636-b1e8-4b58-bdf8-69b7959c00b5 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:14.886815 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1640636-b1e8-4b58-bdf8-69b7959c00b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:14.887263 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1640636-b1e8-4b58-bdf8-69b7959c00b5 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:14.888938 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1640636-b1e8-4b58-bdf8-69b7959c00b5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:14.892374 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1640636-b1e8-4b58-bdf8-69b7959c00b5 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:14.922485 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1312/2607] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:53:14 2022] POST /identity/v3/users => generated 290 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:53:14.927205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-277b83d5-25a7-40c8-b719-6ec3e5396802 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:14.941005 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-277b83d5-25a7-40c8-b719-6ec3e5396802 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:14.946644 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d78635-74b3-42ea-baf3-8684ce0c5e8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:14.947380 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d78635-74b3-42ea-baf3-8684ce0c5e8e admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:14.947715 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d78635-74b3-42ea-baf3-8684ce0c5e8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:14.948049 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58d78635-74b3-42ea-baf3-8684ce0c5e8e admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:14.948827 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58d78635-74b3-42ea-baf3-8684ce0c5e8e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:14.950602 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58d78635-74b3-42ea-baf3-8684ce0c5e8e admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:14.964739 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1296/2608] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:53:14 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:53:14.970710 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1640636-b1e8-4b58-bdf8-69b7959c00b5 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:14.985647 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1640636-b1e8-4b58-bdf8-69b7959c00b5 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:14.991326 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8bad82-25e7-44a6-ab10-6a98a533a890 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:14.992373 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8bad82-25e7-44a6-ab10-6a98a533a890 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:14.992747 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8bad82-25e7-44a6-ab10-6a98a533a890 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:14.993082 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8bad82-25e7-44a6-ab10-6a98a533a890 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:14.994443 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8bad82-25e7-44a6-ab10-6a98a533a890 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:14.996264 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8bad82-25e7-44a6-ab10-6a98a533a890 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:15.013882 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1313/2609] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:53:14 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:53:15.020256 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d78635-74b3-42ea-baf3-8684ce0c5e8e admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:15.033629 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58d78635-74b3-42ea-baf3-8684ce0c5e8e admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:15.040183 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cfaef0d-eb7a-476b-888b-be1446817b63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:15.041013 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfaef0d-eb7a-476b-888b-be1446817b63 admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:15.041386 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfaef0d-eb7a-476b-888b-be1446817b63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:15.041759 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfaef0d-eb7a-476b-888b-be1446817b63 admin admin] PATH_INFO: `/v3/projects/2d9d39b4e4b64318813f48df2bd683ee/users/b6e5581a90c84bbea4c56fab57c72138/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:15.063543 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cfaef0d-eb7a-476b-888b-be1446817b63 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2d9d39b4e4b64318813f48df2bd683ee, user_id=b6e5581a90c84bbea4c56fab57c72138, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:15.067757 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cfaef0d-eb7a-476b-888b-be1446817b63 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:15.089815 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1297/2610] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:53:15 2022] PUT /identity/v3/projects/2d9d39b4e4b64318813f48df2bd683ee/users/b6e5581a90c84bbea4c56fab57c72138/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:15.106321 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c08874-432f-404e-8bf5-d7442588a409 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:15.106789 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c08874-432f-404e-8bf5-d7442588a409 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:15.107996 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54c08874-432f-404e-8bf5-d7442588a409 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:15.154432 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-54c08874-432f-404e-8bf5-d7442588a409 None None] MFA Rules not processed for user `b6e5581a90c84bbea4c56fab57c72138`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:15.157022 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-54c08874-432f-404e-8bf5-d7442588a409 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:15.257334 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:INSERT=4,keystone:SELECT=96 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:15.300286 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1314/2611] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:15 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:53:15.307151 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cfaef0d-eb7a-476b-888b-be1446817b63 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:15.409798 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cfaef0d-eb7a-476b-888b-be1446817b63 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:15.419913 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f54b11-8422-4520-8f1b-174f044199ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:15.421237 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f54b11-8422-4520-8f1b-174f044199ed admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:15.422129 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f54b11-8422-4520-8f1b-174f044199ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:15.422641 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f54b11-8422-4520-8f1b-174f044199ed admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:15.424353 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f54b11-8422-4520-8f1b-174f044199ed admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:15.428308 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f54b11-8422-4520-8f1b-174f044199ed admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:15.472980 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1298/2612] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:53:15 2022] POST /identity/v3/projects => generated 379 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:53:15.478422 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c08874-432f-404e-8bf5-d7442588a409 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:15.497851 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54c08874-432f-404e-8bf5-d7442588a409 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:15.581268 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8eeb5f-821f-4f03-a9ef-b90defdf810a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:15.581997 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8eeb5f-821f-4f03-a9ef-b90defdf810a admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:15.582384 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8eeb5f-821f-4f03-a9ef-b90defdf810a admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:15.582902 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8eeb5f-821f-4f03-a9ef-b90defdf810a admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:15.583684 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a8eeb5f-821f-4f03-a9ef-b90defdf810a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:15.585909 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a8eeb5f-821f-4f03-a9ef-b90defdf810a admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:15.615540 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1315/2613] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:53:15 2022] POST /identity/v3/users => generated 296 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:53:15.620151 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f54b11-8422-4520-8f1b-174f044199ed admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:15.638351 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f54b11-8422-4520-8f1b-174f044199ed admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:15.646673 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1409117-86de-4f2c-8a3c-c23eb5fa6a33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:15.647717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1409117-86de-4f2c-8a3c-c23eb5fa6a33 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:15.648223 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1409117-86de-4f2c-8a3c-c23eb5fa6a33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:15.648706 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1409117-86de-4f2c-8a3c-c23eb5fa6a33 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:15.649919 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1409117-86de-4f2c-8a3c-c23eb5fa6a33 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:15.652568 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1409117-86de-4f2c-8a3c-c23eb5fa6a33 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:15.674071 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1299/2614] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:53:15 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:53:15.678472 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8eeb5f-821f-4f03-a9ef-b90defdf810a admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:15.691783 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8eeb5f-821f-4f03-a9ef-b90defdf810a admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:15.697862 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e30d31e-7b86-42f8-b158-b6f0540dbd7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:15.698796 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e30d31e-7b86-42f8-b158-b6f0540dbd7c admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:15.700076 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e30d31e-7b86-42f8-b158-b6f0540dbd7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:15.700076 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e30d31e-7b86-42f8-b158-b6f0540dbd7c admin admin] PATH_INFO: `/v3/projects/5efc5497692d41979cbafbd469707017/users/c9490602f0ae4f5db3942476725813d7/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:15.724145 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e30d31e-7b86-42f8-b158-b6f0540dbd7c admin admin] RBAC: Authorizing `identity:create_grant(project_id=5efc5497692d41979cbafbd469707017, user_id=c9490602f0ae4f5db3942476725813d7, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:15.727611 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e30d31e-7b86-42f8-b158-b6f0540dbd7c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:15.743514 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1316/2615] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:53:15 2022] PUT /identity/v3/projects/5efc5497692d41979cbafbd469707017/users/c9490602f0ae4f5db3942476725813d7/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:15.748607 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1409117-86de-4f2c-8a3c-c23eb5fa6a33 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:15.766924 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1409117-86de-4f2c-8a3c-c23eb5fa6a33 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:15.849015 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:15.864692 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d26f1d61-e1d5-4580-affd-5edc598448d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:15.865400 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26f1d61-e1d5-4580-affd-5edc598448d7 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:15.865784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26f1d61-e1d5-4580-affd-5edc598448d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:15.866154 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26f1d61-e1d5-4580-affd-5edc598448d7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:15.867077 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d26f1d61-e1d5-4580-affd-5edc598448d7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:15.868979 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d26f1d61-e1d5-4580-affd-5edc598448d7 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:15.880431 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1300/2616] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:53:15 2022] GET /identity/v3/roles => generated 1420 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:53:15.884914 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e30d31e-7b86-42f8-b158-b6f0540dbd7c admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:15.903524 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e30d31e-7b86-42f8-b158-b6f0540dbd7c admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:15.910382 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e153fa9-e023-459f-b9a7-894cfffeaf60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:15.911210 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e153fa9-e023-459f-b9a7-894cfffeaf60 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:15.912189 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e153fa9-e023-459f-b9a7-894cfffeaf60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:15.912189 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e153fa9-e023-459f-b9a7-894cfffeaf60 admin admin] PATH_INFO: `/v3/projects/5efc5497692d41979cbafbd469707017/users/c9490602f0ae4f5db3942476725813d7/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:15.914189 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e153fa9-e023-459f-b9a7-894cfffeaf60 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5efc5497692d41979cbafbd469707017, user_id=c9490602f0ae4f5db3942476725813d7, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:15.916980 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e153fa9-e023-459f-b9a7-894cfffeaf60 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:15.936617 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1317/2617] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:53:15 2022] PUT /identity/v3/projects/5efc5497692d41979cbafbd469707017/users/c9490602f0ae4f5db3942476725813d7/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:53:15.953849 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767f6710-57b8-4bf8-822c-96abeaf959ea None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:15.954482 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767f6710-57b8-4bf8-822c-96abeaf959ea None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:15.954981 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767f6710-57b8-4bf8-822c-96abeaf959ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:15.988412 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-767f6710-57b8-4bf8-822c-96abeaf959ea None None] MFA Rules not processed for user `c9490602f0ae4f5db3942476725813d7`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:15.989676 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-767f6710-57b8-4bf8-822c-96abeaf959ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:16.076671 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1301/2618] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:15 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:53:16.088717 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e153fa9-e023-459f-b9a7-894cfffeaf60 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:16.090868 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-6e153fa9-e023-459f-b9a7-894cfffeaf60 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:16.180210 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e153fa9-e023-459f-b9a7-894cfffeaf60 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:16.188588 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a045f362-df2e-4d95-9d9d-7c49e2fc9ce9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:16.189549 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a045f362-df2e-4d95-9d9d-7c49e2fc9ce9 service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:16.190131 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a045f362-df2e-4d95-9d9d-7c49e2fc9ce9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:16.190676 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a045f362-df2e-4d95-9d9d-7c49e2fc9ce9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:16.209605 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a045f362-df2e-4d95-9d9d-7c49e2fc9ce9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:16.212455 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a045f362-df2e-4d95-9d9d-7c49e2fc9ce9 service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:16.231438 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1318/2619] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:53:16 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:53:16.310946 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767f6710-57b8-4bf8-822c-96abeaf959ea None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:16.326541 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767f6710-57b8-4bf8-822c-96abeaf959ea None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:16.332125 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35820b9-0d3f-40db-bfa7-0ef91458932b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:16.332751 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35820b9-0d3f-40db-bfa7-0ef91458932b service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:16.333029 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35820b9-0d3f-40db-bfa7-0ef91458932b service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:16.333303 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35820b9-0d3f-40db-bfa7-0ef91458932b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:16.460794 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a35820b9-0d3f-40db-bfa7-0ef91458932b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:16.462729 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a35820b9-0d3f-40db-bfa7-0ef91458932b service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:16.474650 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1302/2620] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:53:16 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 165 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:53:17.319610 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a045f362-df2e-4d95-9d9d-7c49e2fc9ce9 service cinder] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:17.322146 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a045f362-df2e-4d95-9d9d-7c49e2fc9ce9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:17.395785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:17.465731 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a045f362-df2e-4d95-9d9d-7c49e2fc9ce9 service cinder] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:17.473811 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c215595b-0992-47b9-812d-dced7299e83c service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3f1d95049d2d4593bf52670b1a0035fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:17.474972 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c215595b-0992-47b9-812d-dced7299e83c service placement] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:17.475353 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c215595b-0992-47b9-812d-dced7299e83c service placement] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:17.475806 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c215595b-0992-47b9-812d-dced7299e83c service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:17.478100 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-c215595b-0992-47b9-812d-dced7299e83c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:17.492783 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c215595b-0992-47b9-812d-dced7299e83c service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:17.494964 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c215595b-0992-47b9-812d-dced7299e83c service placement] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:17.508241 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1319/2621] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:53:17 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 190 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:53:19.968472 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35820b9-0d3f-40db-bfa7-0ef91458932b service cinder] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:20.072123 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35820b9-0d3f-40db-bfa7-0ef91458932b service cinder] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:20.077290 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84ab8ff1-93cf-453d-90c5-97bc5acbc1c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:20.077957 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ab8ff1-93cf-453d-90c5-97bc5acbc1c8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:20.078243 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ab8ff1-93cf-453d-90c5-97bc5acbc1c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:20.078562 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ab8ff1-93cf-453d-90c5-97bc5acbc1c8 admin admin] PATH_INFO: `/v3/users/a8e351115d2b449c9544905c2db91e69` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:20.079927 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84ab8ff1-93cf-453d-90c5-97bc5acbc1c8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a8e351115d2b449c9544905c2db91e69)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:20.082324 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84ab8ff1-93cf-453d-90c5-97bc5acbc1c8 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:20.100069 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:20.154437 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-84ab8ff1-93cf-453d-90c5-97bc5acbc1c8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a8e351115d2b449c9544905c2db91e69'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:20.166372 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-84ab8ff1-93cf-453d-90c5-97bc5acbc1c8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a8e351115d2b449c9544905c2db91e69'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:20.167177 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-84ab8ff1-93cf-453d-90c5-97bc5acbc1c8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a8e351115d2b449c9544905c2db91e69'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:20.167904 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-84ab8ff1-93cf-453d-90c5-97bc5acbc1c8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a8e351115d2b449c9544905c2db91e69'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:20.177328 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-84ab8ff1-93cf-453d-90c5-97bc5acbc1c8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a8e351115d2b449c9544905c2db91e69'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:20.199217 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1303/2622] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:53:19 2022] DELETE /identity/v3/users/a8e351115d2b449c9544905c2db91e69 => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:20.205187 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3ef7e5-1c47-45d8-a0ef-13dc8ef96d72 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:20.206151 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3ef7e5-1c47-45d8-a0ef-13dc8ef96d72 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:20.206151 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3ef7e5-1c47-45d8-a0ef-13dc8ef96d72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:20.229888 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-4d3ef7e5-1c47-45d8-a0ef-13dc8ef96d72 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:20.231200 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4d3ef7e5-1c47-45d8-a0ef-13dc8ef96d72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:20.305466 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1320/2623] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:53:20.321085 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84ab8ff1-93cf-453d-90c5-97bc5acbc1c8 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:20.322805 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-84ab8ff1-93cf-453d-90c5-97bc5acbc1c8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:20.427941 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84ab8ff1-93cf-453d-90c5-97bc5acbc1c8 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:20.432507 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-617ed6a8-693a-4448-bd71-f1463413b09c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:20.433312 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617ed6a8-693a-4448-bd71-f1463413b09c service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:20.433707 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617ed6a8-693a-4448-bd71-f1463413b09c service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:20.434025 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617ed6a8-693a-4448-bd71-f1463413b09c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:20.444559 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-617ed6a8-693a-4448-bd71-f1463413b09c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:20.446031 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-617ed6a8-693a-4448-bd71-f1463413b09c service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:20.457762 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1304/2624] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:53:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:53:20.856001 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d3ef7e5-1c47-45d8-a0ef-13dc8ef96d72 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:20.859535 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4d3ef7e5-1c47-45d8-a0ef-13dc8ef96d72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:20.884785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d3ef7e5-1c47-45d8-a0ef-13dc8ef96d72 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:20.894359 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f77dedf-eb8b-4432-bbdb-c71f1c0b8f1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:20.895654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f77dedf-eb8b-4432-bbdb-c71f1c0b8f1c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:20.896126 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f77dedf-eb8b-4432-bbdb-c71f1c0b8f1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:20.896887 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f77dedf-eb8b-4432-bbdb-c71f1c0b8f1c admin admin] PATH_INFO: `/v3/projects/16939c0c855a44cba5d357baf68bc9be` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:20.899357 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f77dedf-eb8b-4432-bbdb-c71f1c0b8f1c admin admin] RBAC: Authorizing `identity:delete_project(project_id=16939c0c855a44cba5d357baf68bc9be)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:20.903807 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f77dedf-eb8b-4432-bbdb-c71f1c0b8f1c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:20.989695 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7f77dedf-eb8b-4432-bbdb-c71f1c0b8f1c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '16939c0c855a44cba5d357baf68bc9be', 'request_id': 'req-7f77dedf-eb8b-4432-bbdb-c71f1c0b8f1c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:20.998526 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7f77dedf-eb8b-4432-bbdb-c71f1c0b8f1c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '16939c0c855a44cba5d357baf68bc9be', 'request_id': 'req-7f77dedf-eb8b-4432-bbdb-c71f1c0b8f1c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:21.012338 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7f77dedf-eb8b-4432-bbdb-c71f1c0b8f1c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '16939c0c855a44cba5d357baf68bc9be', 'request_id': 'req-7f77dedf-eb8b-4432-bbdb-c71f1c0b8f1c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:21.053764 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7f77dedf-eb8b-4432-bbdb-c71f1c0b8f1c admin admin] The token cache is being invalidate because project 16939c0c855a44cba5d357baf68bc9be was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:53:21.054790 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7f77dedf-eb8b-4432-bbdb-c71f1c0b8f1c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:21.059102 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7f77dedf-eb8b-4432-bbdb-c71f1c0b8f1c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:21.062589 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1321/2625] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:53:20 2022] DELETE /identity/v3/projects/16939c0c855a44cba5d357baf68bc9be => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:53:21.070847 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-617ed6a8-693a-4448-bd71-f1463413b09c service neutron] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:21.093342 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-617ed6a8-693a-4448-bd71-f1463413b09c service neutron] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:21.192770 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62797918-f6ef-42f8-8f1d-d34ce076a40d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:21.193426 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62797918-f6ef-42f8-8f1d-d34ce076a40d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:21.193717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62797918-f6ef-42f8-8f1d-d34ce076a40d admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:21.193992 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62797918-f6ef-42f8-8f1d-d34ce076a40d admin admin] PATH_INFO: `/v3/users/b6e5581a90c84bbea4c56fab57c72138` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:21.195257 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62797918-f6ef-42f8-8f1d-d34ce076a40d admin admin] RBAC: Authorizing `identity:delete_user(user_id=b6e5581a90c84bbea4c56fab57c72138)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:21.197220 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62797918-f6ef-42f8-8f1d-d34ce076a40d admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:21.215164 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:21.268438 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-62797918-f6ef-42f8-8f1d-d34ce076a40d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b6e5581a90c84bbea4c56fab57c72138'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:21.278496 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-62797918-f6ef-42f8-8f1d-d34ce076a40d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b6e5581a90c84bbea4c56fab57c72138'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:21.279749 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-62797918-f6ef-42f8-8f1d-d34ce076a40d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b6e5581a90c84bbea4c56fab57c72138'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:21.280397 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-62797918-f6ef-42f8-8f1d-d34ce076a40d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b6e5581a90c84bbea4c56fab57c72138'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:21.289404 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-62797918-f6ef-42f8-8f1d-d34ce076a40d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b6e5581a90c84bbea4c56fab57c72138'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:21.318219 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1305/2626] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:53:21 2022] DELETE /identity/v3/users/b6e5581a90c84bbea4c56fab57c72138 => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:21.675683 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f77dedf-eb8b-4432-bbdb-c71f1c0b8f1c admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:21.678062 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7f77dedf-eb8b-4432-bbdb-c71f1c0b8f1c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:21.697825 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f77dedf-eb8b-4432-bbdb-c71f1c0b8f1c admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:21.738467 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:21.779152 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3735b9b6-71f9-44ec-a28a-2c890fb5492b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:21.779720 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3735b9b6-71f9-44ec-a28a-2c890fb5492b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:21.780005 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3735b9b6-71f9-44ec-a28a-2c890fb5492b admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:21.780283 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3735b9b6-71f9-44ec-a28a-2c890fb5492b admin admin] PATH_INFO: `/v3/projects/2d9d39b4e4b64318813f48df2bd683ee` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:21.781477 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3735b9b6-71f9-44ec-a28a-2c890fb5492b admin admin] RBAC: Authorizing `identity:delete_project(project_id=2d9d39b4e4b64318813f48df2bd683ee)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:21.783468 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3735b9b6-71f9-44ec-a28a-2c890fb5492b admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:21.840176 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-3735b9b6-71f9-44ec-a28a-2c890fb5492b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2d9d39b4e4b64318813f48df2bd683ee', 'request_id': 'req-3735b9b6-71f9-44ec-a28a-2c890fb5492b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:21.846984 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-3735b9b6-71f9-44ec-a28a-2c890fb5492b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2d9d39b4e4b64318813f48df2bd683ee', 'request_id': 'req-3735b9b6-71f9-44ec-a28a-2c890fb5492b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:21.856193 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-3735b9b6-71f9-44ec-a28a-2c890fb5492b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2d9d39b4e4b64318813f48df2bd683ee', 'request_id': 'req-3735b9b6-71f9-44ec-a28a-2c890fb5492b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:21.868995 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-3735b9b6-71f9-44ec-a28a-2c890fb5492b admin admin] The token cache is being invalidate because project 2d9d39b4e4b64318813f48df2bd683ee was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:53:21.869278 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-3735b9b6-71f9-44ec-a28a-2c890fb5492b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:21.869963 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-3735b9b6-71f9-44ec-a28a-2c890fb5492b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:21.871539 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1322/2627] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:53:21 2022] DELETE /identity/v3/projects/2d9d39b4e4b64318813f48df2bd683ee => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:53:21.876371 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62797918-f6ef-42f8-8f1d-d34ce076a40d admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:21.877964 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-62797918-f6ef-42f8-8f1d-d34ce076a40d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:21.891505 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62797918-f6ef-42f8-8f1d-d34ce076a40d admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:21.987538 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2a2659-74fa-4034-b46b-631c3d68c401 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:21.988629 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2a2659-74fa-4034-b46b-631c3d68c401 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:21.989134 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2a2659-74fa-4034-b46b-631c3d68c401 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:21.989619 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2a2659-74fa-4034-b46b-631c3d68c401 admin admin] PATH_INFO: `/v3/users/c9490602f0ae4f5db3942476725813d7` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:21.991768 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b2a2659-74fa-4034-b46b-631c3d68c401 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c9490602f0ae4f5db3942476725813d7)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:21.996062 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b2a2659-74fa-4034-b46b-631c3d68c401 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:22.078318 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-4b2a2659-74fa-4034-b46b-631c3d68c401 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c9490602f0ae4f5db3942476725813d7'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:22.094300 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-4b2a2659-74fa-4034-b46b-631c3d68c401 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c9490602f0ae4f5db3942476725813d7'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:22.095483 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-4b2a2659-74fa-4034-b46b-631c3d68c401 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c9490602f0ae4f5db3942476725813d7'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:22.096544 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-4b2a2659-74fa-4034-b46b-631c3d68c401 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c9490602f0ae4f5db3942476725813d7'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:22.110199 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-4b2a2659-74fa-4034-b46b-631c3d68c401 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c9490602f0ae4f5db3942476725813d7'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:22.145940 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1306/2628] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:53:21 2022] DELETE /identity/v3/users/c9490602f0ae4f5db3942476725813d7 => generated 0 bytes in 267 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:53:22.506900 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3735b9b6-71f9-44ec-a28a-2c890fb5492b admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:22.507169 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-3735b9b6-71f9-44ec-a28a-2c890fb5492b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:22.560598 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2a2659-74fa-4034-b46b-631c3d68c401 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:22.565171 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-4b2a2659-74fa-4034-b46b-631c3d68c401 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:22.606052 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=79,keystone:DELETE=19,keystone:INSERT=2 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:22.669632 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3735b9b6-71f9-44ec-a28a-2c890fb5492b admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:22.685366 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2a2659-74fa-4034-b46b-631c3d68c401 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:22.693803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8511022-e1e7-4f41-a919-db0a3cc04c2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:22.695035 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8511022-e1e7-4f41-a919-db0a3cc04c2c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:22.695542 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8511022-e1e7-4f41-a919-db0a3cc04c2c admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:22.696089 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8511022-e1e7-4f41-a919-db0a3cc04c2c admin admin] PATH_INFO: `/v3/projects/5efc5497692d41979cbafbd469707017` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:22.698129 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8511022-e1e7-4f41-a919-db0a3cc04c2c admin admin] RBAC: Authorizing `identity:delete_project(project_id=5efc5497692d41979cbafbd469707017)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:22.702591 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8511022-e1e7-4f41-a919-db0a3cc04c2c admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:22.742947 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ba5c1a-7b29-4ea4-b162-7ffe5bfdb179 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:22.743689 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ba5c1a-7b29-4ea4-b162-7ffe5bfdb179 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:22.744045 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ba5c1a-7b29-4ea4-b162-7ffe5bfdb179 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:22.744360 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ba5c1a-7b29-4ea4-b162-7ffe5bfdb179 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:22.746527 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-51ba5c1a-7b29-4ea4-b162-7ffe5bfdb179 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:22.760237 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51ba5c1a-7b29-4ea4-b162-7ffe5bfdb179 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:22.761702 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51ba5c1a-7b29-4ea4-b162-7ffe5bfdb179 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:22.774302 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1323/2629] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:53:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 269 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:53:22.777546 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-e8511022-e1e7-4f41-a919-db0a3cc04c2c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5efc5497692d41979cbafbd469707017', 'request_id': 'req-e8511022-e1e7-4f41-a919-db0a3cc04c2c'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:22.785565 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-e8511022-e1e7-4f41-a919-db0a3cc04c2c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5efc5497692d41979cbafbd469707017', 'request_id': 'req-e8511022-e1e7-4f41-a919-db0a3cc04c2c'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:22.795830 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-e8511022-e1e7-4f41-a919-db0a3cc04c2c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5efc5497692d41979cbafbd469707017', 'request_id': 'req-e8511022-e1e7-4f41-a919-db0a3cc04c2c'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:22.812780 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-e8511022-e1e7-4f41-a919-db0a3cc04c2c admin admin] The token cache is being invalidate because project 5efc5497692d41979cbafbd469707017 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:53:22.813504 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-e8511022-e1e7-4f41-a919-db0a3cc04c2c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:22.814795 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-e8511022-e1e7-4f41-a919-db0a3cc04c2c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:22.817482 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1307/2630] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:53:22 2022] DELETE /identity/v3/projects/5efc5497692d41979cbafbd469707017 => generated 0 bytes in 257 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:22.824557 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eec0720-01f0-4575-8219-3a1d955e1273 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:22.825047 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eec0720-01f0-4575-8219-3a1d955e1273 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:22.825472 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eec0720-01f0-4575-8219-3a1d955e1273 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:22.852354 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-9eec0720-01f0-4575-8219-3a1d955e1273 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:22.853528 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-9eec0720-01f0-4575-8219-3a1d955e1273 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:22.876372 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:22.963130 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1324/2631] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:53:23.064199 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b64e25-fb83-4ec4-9423-f224b000145b None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:23.064693 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b64e25-fb83-4ec4-9423-f224b000145b None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:23.065192 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b64e25-fb83-4ec4-9423-f224b000145b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:23.095330 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-e9b64e25-fb83-4ec4-9423-f224b000145b None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:23.097353 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-e9b64e25-fb83-4ec4-9423-f224b000145b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:23.108061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1308/2632] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:53:23.113886 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eec0720-01f0-4575-8219-3a1d955e1273 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:23.127047 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eec0720-01f0-4575-8219-3a1d955e1273 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:23.133537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a12526-01fe-4d92-8e9d-159e54fe416b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:23.134639 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a12526-01fe-4d92-8e9d-159e54fe416b admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:23.135139 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a12526-01fe-4d92-8e9d-159e54fe416b admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:23.135630 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a12526-01fe-4d92-8e9d-159e54fe416b admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:23.136761 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a12526-01fe-4d92-8e9d-159e54fe416b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:53:23.137286 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a12526-01fe-4d92-8e9d-159e54fe416b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:23.140477 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a12526-01fe-4d92-8e9d-159e54fe416b admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:23.161804 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1325/2633] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:53:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:53:23.167195 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b64e25-fb83-4ec4-9423-f224b000145b None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:23.178955 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b64e25-fb83-4ec4-9423-f224b000145b None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:23.184031 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bcfda7e-f860-4ba8-8f5a-a73380ed4b0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:23.184669 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bcfda7e-f860-4ba8-8f5a-a73380ed4b0c admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:23.185029 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bcfda7e-f860-4ba8-8f5a-a73380ed4b0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:23.185329 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bcfda7e-f860-4ba8-8f5a-a73380ed4b0c admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:23.185996 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bcfda7e-f860-4ba8-8f5a-a73380ed4b0c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:23.188013 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bcfda7e-f860-4ba8-8f5a-a73380ed4b0c admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:23.226191 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1309/2634] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:53:23 2022] POST /identity/v3/projects => generated 385 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:53:23.227963 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a12526-01fe-4d92-8e9d-159e54fe416b admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:23.240003 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a12526-01fe-4d92-8e9d-159e54fe416b admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:23.341937 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e32933c-2f80-4134-a1fb-c549e2cd4492 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:23.342774 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e32933c-2f80-4134-a1fb-c549e2cd4492 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:23.343163 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e32933c-2f80-4134-a1fb-c549e2cd4492 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:23.343580 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e32933c-2f80-4134-a1fb-c549e2cd4492 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:23.344801 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e32933c-2f80-4134-a1fb-c549e2cd4492 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:23.347423 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e32933c-2f80-4134-a1fb-c549e2cd4492 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:23.374898 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1326/2635] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:53:23 2022] POST /identity/v3/users => generated 293 bytes in 147 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:53:23.379419 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bcfda7e-f860-4ba8-8f5a-a73380ed4b0c admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:23.391994 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bcfda7e-f860-4ba8-8f5a-a73380ed4b0c admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:23.400405 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b56c6d-42d6-47a8-a8f0-7a77355cb002 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:23.401689 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b56c6d-42d6-47a8-a8f0-7a77355cb002 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:23.402183 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b56c6d-42d6-47a8-a8f0-7a77355cb002 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:23.402746 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b56c6d-42d6-47a8-a8f0-7a77355cb002 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:23.404776 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2b56c6d-42d6-47a8-a8f0-7a77355cb002 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:23.407424 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2b56c6d-42d6-47a8-a8f0-7a77355cb002 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:23.434554 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1310/2636] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:53:23 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:53:23.439968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e32933c-2f80-4134-a1fb-c549e2cd4492 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:23.460417 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e32933c-2f80-4134-a1fb-c549e2cd4492 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:23.469423 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ba6b1a-9cd4-4b26-b01a-7280d3975f10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:23.470573 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ba6b1a-9cd4-4b26-b01a-7280d3975f10 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:23.471146 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ba6b1a-9cd4-4b26-b01a-7280d3975f10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:23.471644 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ba6b1a-9cd4-4b26-b01a-7280d3975f10 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:23.472896 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57ba6b1a-9cd4-4b26-b01a-7280d3975f10 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:23.476085 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57ba6b1a-9cd4-4b26-b01a-7280d3975f10 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:23.494983 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1327/2637] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:53:23 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:53:23.499487 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b56c6d-42d6-47a8-a8f0-7a77355cb002 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:23.510678 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b56c6d-42d6-47a8-a8f0-7a77355cb002 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:23.516483 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c5d514-243e-4828-8975-000c1ab338da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:23.517179 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c5d514-243e-4828-8975-000c1ab338da admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:23.517481 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c5d514-243e-4828-8975-000c1ab338da admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:23.517744 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c5d514-243e-4828-8975-000c1ab338da admin admin] PATH_INFO: `/v3/projects/3324026c0b3f466aa55c727eb812aa15/users/290e0d5cdf2348eead0e1c10d066d817/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:23.536365 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84c5d514-243e-4828-8975-000c1ab338da admin admin] RBAC: Authorizing `identity:create_grant(project_id=3324026c0b3f466aa55c727eb812aa15, user_id=290e0d5cdf2348eead0e1c10d066d817, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:23.539685 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84c5d514-243e-4828-8975-000c1ab338da admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:23.559639 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1311/2638] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:53:23 2022] PUT /identity/v3/projects/3324026c0b3f466aa55c727eb812aa15/users/290e0d5cdf2348eead0e1c10d066d817/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:53:23.573375 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd181114-ed62-4884-8f43-5fc7dff879e0 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:23.573730 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd181114-ed62-4884-8f43-5fc7dff879e0 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:23.574210 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd181114-ed62-4884-8f43-5fc7dff879e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:23.624216 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-bd181114-ed62-4884-8f43-5fc7dff879e0 None None] MFA Rules not processed for user `290e0d5cdf2348eead0e1c10d066d817`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:23.626666 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-bd181114-ed62-4884-8f43-5fc7dff879e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:23.746849 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1328/2639] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:23 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:53:23.755276 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c5d514-243e-4828-8975-000c1ab338da admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:23.833394 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:23.883897 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c5d514-243e-4828-8975-000c1ab338da admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:23.893551 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d5c00f-0852-40cf-a40f-5fb7a763b8f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:23.894678 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d5c00f-0852-40cf-a40f-5fb7a763b8f3 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:23.895083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d5c00f-0852-40cf-a40f-5fb7a763b8f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:23.895463 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d5c00f-0852-40cf-a40f-5fb7a763b8f3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:23.896282 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9d5c00f-0852-40cf-a40f-5fb7a763b8f3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:23.898734 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9d5c00f-0852-40cf-a40f-5fb7a763b8f3 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:23.936648 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1312/2640] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:53:23 2022] POST /identity/v3/projects => generated 387 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:53:23.941175 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd181114-ed62-4884-8f43-5fc7dff879e0 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:23.957621 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd181114-ed62-4884-8f43-5fc7dff879e0 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:23.973531 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:24.045506 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6bd4d89-e85d-4409-9895-90b8dde0acb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:24.046298 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bd4d89-e85d-4409-9895-90b8dde0acb9 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:24.046606 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bd4d89-e85d-4409-9895-90b8dde0acb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:24.047320 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bd4d89-e85d-4409-9895-90b8dde0acb9 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:24.048204 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6bd4d89-e85d-4409-9895-90b8dde0acb9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:24.050480 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6bd4d89-e85d-4409-9895-90b8dde0acb9 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:24.090977 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1329/2641] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:53:23 2022] POST /identity/v3/users => generated 300 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:53:24.106290 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d5c00f-0852-40cf-a40f-5fb7a763b8f3 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:24.126761 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9d5c00f-0852-40cf-a40f-5fb7a763b8f3 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:24.135423 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-710364bf-1f14-4541-b4f4-462aad81b05f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:24.137073 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710364bf-1f14-4541-b4f4-462aad81b05f admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:24.137909 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710364bf-1f14-4541-b4f4-462aad81b05f admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:24.138856 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710364bf-1f14-4541-b4f4-462aad81b05f admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:24.140467 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-710364bf-1f14-4541-b4f4-462aad81b05f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:24.144309 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-710364bf-1f14-4541-b4f4-462aad81b05f admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:24.179722 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1313/2642] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:53:24 2022] GET /identity/v3/roles => generated 1420 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:53:24.181701 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6bd4d89-e85d-4409-9895-90b8dde0acb9 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:24.206065 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6bd4d89-e85d-4409-9895-90b8dde0acb9 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:24.221300 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db05ae49-878b-4bc8-b53c-9eaf998d13a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:24.222561 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db05ae49-878b-4bc8-b53c-9eaf998d13a0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:24.223050 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db05ae49-878b-4bc8-b53c-9eaf998d13a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:24.223760 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db05ae49-878b-4bc8-b53c-9eaf998d13a0 admin admin] PATH_INFO: `/v3/projects/42fd5b19bfc842888f6b3b554b173ed7/users/26ad59a7c25948f69fe011cf6cd9804b/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:24.252440 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db05ae49-878b-4bc8-b53c-9eaf998d13a0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=42fd5b19bfc842888f6b3b554b173ed7, user_id=26ad59a7c25948f69fe011cf6cd9804b, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:24.255510 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db05ae49-878b-4bc8-b53c-9eaf998d13a0 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:24.275985 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1330/2643] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:53:24 2022] PUT /identity/v3/projects/42fd5b19bfc842888f6b3b554b173ed7/users/26ad59a7c25948f69fe011cf6cd9804b/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:24.280689 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-710364bf-1f14-4541-b4f4-462aad81b05f admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:24.292736 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-710364bf-1f14-4541-b4f4-462aad81b05f admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:24.402419 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec9b95c-e9ed-4170-bcde-075d41eda1da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:24.403840 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec9b95c-e9ed-4170-bcde-075d41eda1da admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:24.404157 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec9b95c-e9ed-4170-bcde-075d41eda1da admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:24.405059 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec9b95c-e9ed-4170-bcde-075d41eda1da admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:24.405864 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec9b95c-e9ed-4170-bcde-075d41eda1da admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:24.407764 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec9b95c-e9ed-4170-bcde-075d41eda1da admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:24.426127 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1314/2644] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:53:24 2022] GET /identity/v3/roles => generated 1420 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:53:24.427699 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db05ae49-878b-4bc8-b53c-9eaf998d13a0 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:24.440668 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db05ae49-878b-4bc8-b53c-9eaf998d13a0 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:24.447449 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241c4f0e-2c1d-4c5e-9f5e-4959ad2695a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:24.448314 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241c4f0e-2c1d-4c5e-9f5e-4959ad2695a0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:24.448699 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241c4f0e-2c1d-4c5e-9f5e-4959ad2695a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:24.450134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241c4f0e-2c1d-4c5e-9f5e-4959ad2695a0 admin admin] PATH_INFO: `/v3/projects/42fd5b19bfc842888f6b3b554b173ed7/users/26ad59a7c25948f69fe011cf6cd9804b/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:24.458753 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-241c4f0e-2c1d-4c5e-9f5e-4959ad2695a0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=42fd5b19bfc842888f6b3b554b173ed7, user_id=26ad59a7c25948f69fe011cf6cd9804b, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:24.463506 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-241c4f0e-2c1d-4c5e-9f5e-4959ad2695a0 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:24.483139 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1331/2645] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:53:24 2022] PUT /identity/v3/projects/42fd5b19bfc842888f6b3b554b173ed7/users/26ad59a7c25948f69fe011cf6cd9804b/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:24.495422 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89297ac0-be32-4061-90ad-1f957da747c5 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:24.495831 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89297ac0-be32-4061-90ad-1f957da747c5 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:24.496210 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89297ac0-be32-4061-90ad-1f957da747c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:24.542735 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-89297ac0-be32-4061-90ad-1f957da747c5 None None] MFA Rules not processed for user `26ad59a7c25948f69fe011cf6cd9804b`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:24.545353 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-89297ac0-be32-4061-90ad-1f957da747c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:24.692763 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1315/2646] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:24 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:53:24.700791 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241c4f0e-2c1d-4c5e-9f5e-4959ad2695a0 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:24.704827 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-241c4f0e-2c1d-4c5e-9f5e-4959ad2695a0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:24.875439 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241c4f0e-2c1d-4c5e-9f5e-4959ad2695a0 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:24.880698 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ada7961-d56c-4157-994b-0bf6f9acdb10 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:24.881762 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ada7961-d56c-4157-994b-0bf6f9acdb10 service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:24.882226 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ada7961-d56c-4157-994b-0bf6f9acdb10 service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:24.882651 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ada7961-d56c-4157-994b-0bf6f9acdb10 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:24.897117 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ada7961-d56c-4157-994b-0bf6f9acdb10 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:24.900010 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ada7961-d56c-4157-994b-0bf6f9acdb10 service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:24.921772 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1332/2647] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:53:24 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 222 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:53:25.930480 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89297ac0-be32-4061-90ad-1f957da747c5 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:26.001902 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:26.070859 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89297ac0-be32-4061-90ad-1f957da747c5 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:26.077181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab5db181-2cb2-4276-85ec-1bcbdfe2f57a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:26.077855 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5db181-2cb2-4276-85ec-1bcbdfe2f57a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:26.078134 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5db181-2cb2-4276-85ec-1bcbdfe2f57a admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:26.079864 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5db181-2cb2-4276-85ec-1bcbdfe2f57a admin admin] PATH_INFO: `/v3/users/290e0d5cdf2348eead0e1c10d066d817` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:26.081165 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab5db181-2cb2-4276-85ec-1bcbdfe2f57a admin admin] RBAC: Authorizing `identity:delete_user(user_id=290e0d5cdf2348eead0e1c10d066d817)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:26.083287 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab5db181-2cb2-4276-85ec-1bcbdfe2f57a admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:26.171685 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-ab5db181-2cb2-4276-85ec-1bcbdfe2f57a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '290e0d5cdf2348eead0e1c10d066d817'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:26.186122 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-ab5db181-2cb2-4276-85ec-1bcbdfe2f57a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '290e0d5cdf2348eead0e1c10d066d817'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:26.187040 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-ab5db181-2cb2-4276-85ec-1bcbdfe2f57a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '290e0d5cdf2348eead0e1c10d066d817'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:26.187801 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-ab5db181-2cb2-4276-85ec-1bcbdfe2f57a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '290e0d5cdf2348eead0e1c10d066d817'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:26.198725 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-ab5db181-2cb2-4276-85ec-1bcbdfe2f57a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '290e0d5cdf2348eead0e1c10d066d817'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:26.224209 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1316/2648] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:53:25 2022] DELETE /identity/v3/users/290e0d5cdf2348eead0e1c10d066d817 => generated 0 bytes in 294 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:53:26.229356 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba62b6fa-9bd0-4ea5-b6b3-9571949af939 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:26.230615 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba62b6fa-9bd0-4ea5-b6b3-9571949af939 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:26.231647 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba62b6fa-9bd0-4ea5-b6b3-9571949af939 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:26.266567 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-ba62b6fa-9bd0-4ea5-b6b3-9571949af939 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:26.266567 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ba62b6fa-9bd0-4ea5-b6b3-9571949af939 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:26.380529 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:26.415958 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1333/2649] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:53:26.437482 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab5db181-2cb2-4276-85ec-1bcbdfe2f57a admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:26.439612 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-ab5db181-2cb2-4276-85ec-1bcbdfe2f57a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:26.600599 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab5db181-2cb2-4276-85ec-1bcbdfe2f57a admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:26.609408 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac84930-1aea-49ba-81cc-76c461126408 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:26.610629 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac84930-1aea-49ba-81cc-76c461126408 service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:26.611548 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac84930-1aea-49ba-81cc-76c461126408 service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:26.612137 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac84930-1aea-49ba-81cc-76c461126408 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:26.631680 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cac84930-1aea-49ba-81cc-76c461126408 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:26.634536 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cac84930-1aea-49ba-81cc-76c461126408 service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:26.656644 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1317/2650] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:53:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 221 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:53:27.294450 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba62b6fa-9bd0-4ea5-b6b3-9571949af939 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:27.300655 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ba62b6fa-9bd0-4ea5-b6b3-9571949af939 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:27.319176 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba62b6fa-9bd0-4ea5-b6b3-9571949af939 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:27.336111 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e4d1ba-6a49-4493-b425-8bf6fccd9f9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:27.336805 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e4d1ba-6a49-4493-b425-8bf6fccd9f9e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:27.337055 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e4d1ba-6a49-4493-b425-8bf6fccd9f9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:27.337402 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e4d1ba-6a49-4493-b425-8bf6fccd9f9e admin admin] PATH_INFO: `/v3/projects/3324026c0b3f466aa55c727eb812aa15` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:27.338631 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e4d1ba-6a49-4493-b425-8bf6fccd9f9e admin admin] RBAC: Authorizing `identity:delete_project(project_id=3324026c0b3f466aa55c727eb812aa15)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:27.340789 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e4d1ba-6a49-4493-b425-8bf6fccd9f9e admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:27.455219 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e0e4d1ba-6a49-4493-b425-8bf6fccd9f9e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3324026c0b3f466aa55c727eb812aa15', 'request_id': 'req-e0e4d1ba-6a49-4493-b425-8bf6fccd9f9e'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:27.473014 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e0e4d1ba-6a49-4493-b425-8bf6fccd9f9e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3324026c0b3f466aa55c727eb812aa15', 'request_id': 'req-e0e4d1ba-6a49-4493-b425-8bf6fccd9f9e'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:27.485885 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e0e4d1ba-6a49-4493-b425-8bf6fccd9f9e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3324026c0b3f466aa55c727eb812aa15', 'request_id': 'req-e0e4d1ba-6a49-4493-b425-8bf6fccd9f9e'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:27.504980 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e0e4d1ba-6a49-4493-b425-8bf6fccd9f9e admin admin] The token cache is being invalidate because project 3324026c0b3f466aa55c727eb812aa15 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:53:27.505348 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e0e4d1ba-6a49-4493-b425-8bf6fccd9f9e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:27.506095 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e0e4d1ba-6a49-4493-b425-8bf6fccd9f9e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:27.511195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1334/2651] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:53:27 2022] DELETE /identity/v3/projects/3324026c0b3f466aa55c727eb812aa15 => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:27.512145 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac84930-1aea-49ba-81cc-76c461126408 service neutron] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:27.531331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac84930-1aea-49ba-81cc-76c461126408 service neutron] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:27.626119 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:27.653613 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd7ca18-2cad-4d9d-b9ab-ab4fde55e226 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:27.654351 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd7ca18-2cad-4d9d-b9ab-ab4fde55e226 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:27.654735 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd7ca18-2cad-4d9d-b9ab-ab4fde55e226 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:27.655010 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd7ca18-2cad-4d9d-b9ab-ab4fde55e226 admin admin] PATH_INFO: `/v3/users/26ad59a7c25948f69fe011cf6cd9804b` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:27.657079 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd7ca18-2cad-4d9d-b9ab-ab4fde55e226 admin admin] RBAC: Authorizing `identity:delete_user(user_id=26ad59a7c25948f69fe011cf6cd9804b)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:27.658286 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd7ca18-2cad-4d9d-b9ab-ab4fde55e226 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:27.680748 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e4d1ba-6a49-4493-b425-8bf6fccd9f9e admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:27.684023 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e0e4d1ba-6a49-4493-b425-8bf6fccd9f9e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:27.716158 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e4d1ba-6a49-4493-b425-8bf6fccd9f9e admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:27.733252 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf8a250-d4ba-464b-845a-47a3a3da02d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:27.734520 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf8a250-d4ba-464b-845a-47a3a3da02d5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:27.735006 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf8a250-d4ba-464b-845a-47a3a3da02d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:27.735451 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf8a250-d4ba-464b-845a-47a3a3da02d5 admin admin] PATH_INFO: `/v3/users/715a782e5a56431e88e4fe5a3b0a62aa` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:27.737087 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf8a250-d4ba-464b-845a-47a3a3da02d5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=715a782e5a56431e88e4fe5a3b0a62aa)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:27.739452 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf8a250-d4ba-464b-845a-47a3a3da02d5 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:27.743554 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-6dd7ca18-2cad-4d9d-b9ab-ab4fde55e226 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '26ad59a7c25948f69fe011cf6cd9804b'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:27.756111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-6dd7ca18-2cad-4d9d-b9ab-ab4fde55e226 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '26ad59a7c25948f69fe011cf6cd9804b'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:27.758357 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-6dd7ca18-2cad-4d9d-b9ab-ab4fde55e226 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '26ad59a7c25948f69fe011cf6cd9804b'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:27.759373 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-6dd7ca18-2cad-4d9d-b9ab-ab4fde55e226 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '26ad59a7c25948f69fe011cf6cd9804b'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:27.771118 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-6dd7ca18-2cad-4d9d-b9ab-ab4fde55e226 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '26ad59a7c25948f69fe011cf6cd9804b'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:27.813513 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1318/2652] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:53:27 2022] DELETE /identity/v3/users/26ad59a7c25948f69fe011cf6cd9804b => generated 0 bytes in 298 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:53:27.836390 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-0bf8a250-d4ba-464b-845a-47a3a3da02d5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '715a782e5a56431e88e4fe5a3b0a62aa'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:27.847254 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-0bf8a250-d4ba-464b-845a-47a3a3da02d5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '715a782e5a56431e88e4fe5a3b0a62aa'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:27.848358 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-0bf8a250-d4ba-464b-845a-47a3a3da02d5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '715a782e5a56431e88e4fe5a3b0a62aa'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:27.849351 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-0bf8a250-d4ba-464b-845a-47a3a3da02d5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '715a782e5a56431e88e4fe5a3b0a62aa'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:27.861433 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-0bf8a250-d4ba-464b-845a-47a3a3da02d5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '715a782e5a56431e88e4fe5a3b0a62aa'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:27.892643 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1335/2653] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:53:27 2022] DELETE /identity/v3/users/715a782e5a56431e88e4fe5a3b0a62aa => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:28.084224 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd7ca18-2cad-4d9d-b9ab-ab4fde55e226 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:28.086852 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-6dd7ca18-2cad-4d9d-b9ab-ab4fde55e226 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:28.188081 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd7ca18-2cad-4d9d-b9ab-ab4fde55e226 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:28.250157 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf8a250-d4ba-464b-845a-47a3a3da02d5 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:28.253060 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-0bf8a250-d4ba-464b-845a-47a3a3da02d5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:28.303390 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5cfdef4-2e4e-4c9d-969b-b2b577f02a27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:28.307276 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5cfdef4-2e4e-4c9d-969b-b2b577f02a27 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:28.307734 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5cfdef4-2e4e-4c9d-969b-b2b577f02a27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:28.308092 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5cfdef4-2e4e-4c9d-969b-b2b577f02a27 admin admin] PATH_INFO: `/v3/users/03af8e89b8434a038fe593252633de19` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:28.309709 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5cfdef4-2e4e-4c9d-969b-b2b577f02a27 admin admin] RBAC: Authorizing `identity:delete_user(user_id=03af8e89b8434a038fe593252633de19)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:28.312660 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5cfdef4-2e4e-4c9d-969b-b2b577f02a27 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:28.352373 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:28.370361 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:28.404628 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf8a250-d4ba-464b-845a-47a3a3da02d5 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:28.411958 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0f1fce-954a-4ffc-a642-846a966ad89d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:28.413001 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0f1fce-954a-4ffc-a642-846a966ad89d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:28.413466 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0f1fce-954a-4ffc-a642-846a966ad89d admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:28.413928 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0f1fce-954a-4ffc-a642-846a966ad89d admin admin] PATH_INFO: `/v3/projects/29c47b13a2044939b5446ffe5397633a` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:28.416056 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd0f1fce-954a-4ffc-a642-846a966ad89d admin admin] RBAC: Authorizing `identity:delete_project(project_id=29c47b13a2044939b5446ffe5397633a)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:28.429746 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd0f1fce-954a-4ffc-a642-846a966ad89d admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:28.440737 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-e5cfdef4-2e4e-4c9d-969b-b2b577f02a27 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '03af8e89b8434a038fe593252633de19'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:28.452819 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-e5cfdef4-2e4e-4c9d-969b-b2b577f02a27 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '03af8e89b8434a038fe593252633de19'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:28.453555 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-e5cfdef4-2e4e-4c9d-969b-b2b577f02a27 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '03af8e89b8434a038fe593252633de19'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:28.454289 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-e5cfdef4-2e4e-4c9d-969b-b2b577f02a27 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '03af8e89b8434a038fe593252633de19'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:28.464047 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-e5cfdef4-2e4e-4c9d-969b-b2b577f02a27 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '03af8e89b8434a038fe593252633de19'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:28.492566 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1319/2654] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:53:28 2022] DELETE /identity/v3/users/03af8e89b8434a038fe593252633de19 => generated 0 bytes in 409 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:28.494092 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5cfdef4-2e4e-4c9d-969b-b2b577f02a27 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:28.496483 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-e5cfdef4-2e4e-4c9d-969b-b2b577f02a27 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:28.512162 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-cd0f1fce-954a-4ffc-a642-846a966ad89d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '29c47b13a2044939b5446ffe5397633a', 'request_id': 'req-cd0f1fce-954a-4ffc-a642-846a966ad89d'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:28.519902 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-cd0f1fce-954a-4ffc-a642-846a966ad89d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '29c47b13a2044939b5446ffe5397633a', 'request_id': 'req-cd0f1fce-954a-4ffc-a642-846a966ad89d'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:28.533514 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-cd0f1fce-954a-4ffc-a642-846a966ad89d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '29c47b13a2044939b5446ffe5397633a', 'request_id': 'req-cd0f1fce-954a-4ffc-a642-846a966ad89d'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:28.559799 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-cd0f1fce-954a-4ffc-a642-846a966ad89d admin admin] The token cache is being invalidate because project 29c47b13a2044939b5446ffe5397633a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:53:28.560493 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-cd0f1fce-954a-4ffc-a642-846a966ad89d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:28.561785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-cd0f1fce-954a-4ffc-a642-846a966ad89d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:28.565137 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1336/2655] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:53:28 2022] DELETE /identity/v3/projects/29c47b13a2044939b5446ffe5397633a => generated 0 bytes in 316 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:28.573025 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ebf7af-754c-4071-862a-60a3f858e1ba None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:28.574037 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ebf7af-754c-4071-862a-60a3f858e1ba None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:28.577873 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ebf7af-754c-4071-862a-60a3f858e1ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:28.636783 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-39ebf7af-754c-4071-862a-60a3f858e1ba None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:28.639495 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-39ebf7af-754c-4071-862a-60a3f858e1ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:28.645691 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5cfdef4-2e4e-4c9d-969b-b2b577f02a27 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:28.652197 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1337/2656] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:53:28.657541 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb81ade4-1dcf-4b61-9367-11f79d4a6164 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:28.658645 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb81ade4-1dcf-4b61-9367-11f79d4a6164 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:28.659136 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb81ade4-1dcf-4b61-9367-11f79d4a6164 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:28.659621 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb81ade4-1dcf-4b61-9367-11f79d4a6164 admin admin] PATH_INFO: `/v3/projects/42fd5b19bfc842888f6b3b554b173ed7` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:28.661794 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb81ade4-1dcf-4b61-9367-11f79d4a6164 admin admin] RBAC: Authorizing `identity:delete_project(project_id=42fd5b19bfc842888f6b3b554b173ed7)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:28.665318 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb81ade4-1dcf-4b61-9367-11f79d4a6164 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:28.666322 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0603b2e-9b6b-4467-8e97-44739d0e8408 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:28.666819 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0603b2e-9b6b-4467-8e97-44739d0e8408 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:28.667280 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0603b2e-9b6b-4467-8e97-44739d0e8408 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:28.717278 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-b0603b2e-9b6b-4467-8e97-44739d0e8408 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:28.720422 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-b0603b2e-9b6b-4467-8e97-44739d0e8408 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:28.748976 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1338/2657] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:53:28.751093 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0603b2e-9b6b-4467-8e97-44739d0e8408 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:28.767021 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0603b2e-9b6b-4467-8e97-44739d0e8408 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:28.776007 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-bb81ade4-1dcf-4b61-9367-11f79d4a6164 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '42fd5b19bfc842888f6b3b554b173ed7', 'request_id': 'req-bb81ade4-1dcf-4b61-9367-11f79d4a6164'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:28.788366 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-bb81ade4-1dcf-4b61-9367-11f79d4a6164 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '42fd5b19bfc842888f6b3b554b173ed7', 'request_id': 'req-bb81ade4-1dcf-4b61-9367-11f79d4a6164'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:28.805255 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-bb81ade4-1dcf-4b61-9367-11f79d4a6164 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '42fd5b19bfc842888f6b3b554b173ed7', 'request_id': 'req-bb81ade4-1dcf-4b61-9367-11f79d4a6164'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:28.822881 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-bb81ade4-1dcf-4b61-9367-11f79d4a6164 admin admin] The token cache is being invalidate because project 42fd5b19bfc842888f6b3b554b173ed7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:53:28.823584 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-bb81ade4-1dcf-4b61-9367-11f79d4a6164 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:28.824855 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-bb81ade4-1dcf-4b61-9367-11f79d4a6164 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:28.826899 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1320/2658] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:53:28 2022] DELETE /identity/v3/projects/42fd5b19bfc842888f6b3b554b173ed7 => generated 0 bytes in 334 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:28.832785 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6214642c-e92b-4909-b4f8-2a63377dead6 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:28.833216 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6214642c-e92b-4909-b4f8-2a63377dead6 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:28.834041 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6214642c-e92b-4909-b4f8-2a63377dead6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:28.853449 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-663b182e-db62-472b-88ad-bbf89ebadf93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:28.854646 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663b182e-db62-472b-88ad-bbf89ebadf93 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:28.855261 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663b182e-db62-472b-88ad-bbf89ebadf93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:28.855575 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663b182e-db62-472b-88ad-bbf89ebadf93 admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:28.857910 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-663b182e-db62-472b-88ad-bbf89ebadf93 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:53:28.858674 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-663b182e-db62-472b-88ad-bbf89ebadf93 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:28.861945 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-663b182e-db62-472b-88ad-bbf89ebadf93 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:28.869805 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-6214642c-e92b-4909-b4f8-2a63377dead6 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:28.873469 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-6214642c-e92b-4909-b4f8-2a63377dead6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:28.885998 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1321/2659] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 54 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:53:28.889849 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1339/2660] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:53:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:53:28.896200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468e63fd-d8be-459c-ac02-53476484533e None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:28.896650 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-663b182e-db62-472b-88ad-bbf89ebadf93 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:28.896835 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468e63fd-d8be-459c-ac02-53476484533e None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:28.897165 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468e63fd-d8be-459c-ac02-53476484533e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:28.900145 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-663b182e-db62-472b-88ad-bbf89ebadf93 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:28.926671 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-663b182e-db62-472b-88ad-bbf89ebadf93 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:28.928188 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-468e63fd-d8be-459c-ac02-53476484533e None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:28.929432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-468e63fd-d8be-459c-ac02-53476484533e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:28.937246 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db7a802-5450-442b-9a45-06f47f37ffe7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:28.938440 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db7a802-5450-442b-9a45-06f47f37ffe7 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:28.939225 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db7a802-5450-442b-9a45-06f47f37ffe7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:28.939820 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db7a802-5450-442b-9a45-06f47f37ffe7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:28.941663 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1322/2661] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:53:28.941908 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3db7a802-5450-442b-9a45-06f47f37ffe7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:28.946296 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-468e63fd-d8be-459c-ac02-53476484533e None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:28.952140 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3db7a802-5450-442b-9a45-06f47f37ffe7 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:28.962230 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-468e63fd-d8be-459c-ac02-53476484533e None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:28.968809 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f522351-7992-41db-8318-f16c4b92ba79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:28.969533 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f522351-7992-41db-8318-f16c4b92ba79 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:28.969874 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f522351-7992-41db-8318-f16c4b92ba79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:28.970174 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f522351-7992-41db-8318-f16c4b92ba79 admin admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:28.970945 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f522351-7992-41db-8318-f16c4b92ba79 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:53:28.971427 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f522351-7992-41db-8318-f16c4b92ba79 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:28.974106 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f522351-7992-41db-8318-f16c4b92ba79 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:28.999005 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1323/2662] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:53:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:53:29.000395 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f522351-7992-41db-8318-f16c4b92ba79 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:29.002705 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-4f522351-7992-41db-8318-f16c4b92ba79 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:29.024302 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:29.035312 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1340/2663] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:53:28 2022] POST /identity/v3/projects => generated 367 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:53:29.035312 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db7a802-5450-442b-9a45-06f47f37ffe7 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:29.035836 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f522351-7992-41db-8318-f16c4b92ba79 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:29.116729 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:29.149854 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c49914e-4f92-4b29-b6b4-c238fa2c1a71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:29.150770 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c49914e-4f92-4b29-b6b4-c238fa2c1a71 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:29.151396 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c49914e-4f92-4b29-b6b4-c238fa2c1a71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:29.151699 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c49914e-4f92-4b29-b6b4-c238fa2c1a71 admin admin] PATH_INFO: `/v3/projects/723d60ae52dc4b76ac83b79e46d8950f` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:29.154513 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c49914e-4f92-4b29-b6b4-c238fa2c1a71 admin admin] RBAC: Authorizing `identity:delete_project(project_id=723d60ae52dc4b76ac83b79e46d8950f)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:29.158431 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c49914e-4f92-4b29-b6b4-c238fa2c1a71 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:29.182322 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db7a802-5450-442b-9a45-06f47f37ffe7 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:29.190954 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c2564a-e8dd-49d9-96f8-ca84a7ed9c84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:29.192015 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c2564a-e8dd-49d9-96f8-ca84a7ed9c84 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:29.192501 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c2564a-e8dd-49d9-96f8-ca84a7ed9c84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:29.193299 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c2564a-e8dd-49d9-96f8-ca84a7ed9c84 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:29.194846 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85c2564a-e8dd-49d9-96f8-ca84a7ed9c84 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:29.197540 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85c2564a-e8dd-49d9-96f8-ca84a7ed9c84 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:29.226479 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-7c49914e-4f92-4b29-b6b4-c238fa2c1a71 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '723d60ae52dc4b76ac83b79e46d8950f', 'request_id': 'req-7c49914e-4f92-4b29-b6b4-c238fa2c1a71'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:29.239437 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-7c49914e-4f92-4b29-b6b4-c238fa2c1a71 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '723d60ae52dc4b76ac83b79e46d8950f', 'request_id': 'req-7c49914e-4f92-4b29-b6b4-c238fa2c1a71'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:29.243354 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1341/2664] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:53:29 2022] POST /identity/v3/projects => generated 375 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:53:29.246005 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c2564a-e8dd-49d9-96f8-ca84a7ed9c84 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:29.253363 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-7c49914e-4f92-4b29-b6b4-c238fa2c1a71 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '723d60ae52dc4b76ac83b79e46d8950f', 'request_id': 'req-7c49914e-4f92-4b29-b6b4-c238fa2c1a71'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:29.269857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-7c49914e-4f92-4b29-b6b4-c238fa2c1a71 admin admin] The token cache is being invalidate because project 723d60ae52dc4b76ac83b79e46d8950f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:53:29.270166 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-7c49914e-4f92-4b29-b6b4-c238fa2c1a71 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:29.271047 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-7c49914e-4f92-4b29-b6b4-c238fa2c1a71 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:29.273059 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1324/2665] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:53:28 2022] DELETE /identity/v3/projects/723d60ae52dc4b76ac83b79e46d8950f => generated 0 bytes in 273 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:29.274312 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c49914e-4f92-4b29-b6b4-c238fa2c1a71 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:29.276588 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-7c49914e-4f92-4b29-b6b4-c238fa2c1a71 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:29.362022 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c2564a-e8dd-49d9-96f8-ca84a7ed9c84 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:29.369165 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aaf022e-306e-4003-83f1-0f1ee5c60776 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:29.369924 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aaf022e-306e-4003-83f1-0f1ee5c60776 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:29.370347 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aaf022e-306e-4003-83f1-0f1ee5c60776 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:29.370776 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aaf022e-306e-4003-83f1-0f1ee5c60776 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:29.371524 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aaf022e-306e-4003-83f1-0f1ee5c60776 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:29.374478 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aaf022e-306e-4003-83f1-0f1ee5c60776 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:29.382717 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c49914e-4f92-4b29-b6b4-c238fa2c1a71 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:29.390680 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98d980d5-9cbf-4f7d-8910-d574b285eb07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:29.391856 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d980d5-9cbf-4f7d-8910-d574b285eb07 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:29.392236 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d980d5-9cbf-4f7d-8910-d574b285eb07 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:29.392779 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98d980d5-9cbf-4f7d-8910-d574b285eb07 admin admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:29.393816 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98d980d5-9cbf-4f7d-8910-d574b285eb07 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:29.396864 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98d980d5-9cbf-4f7d-8910-d574b285eb07 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:29.405056 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1342/2666] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:53:29 2022] POST /identity/v3/users => generated 284 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:53:29.406293 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aaf022e-306e-4003-83f1-0f1ee5c60776 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:29.408474 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-7aaf022e-306e-4003-83f1-0f1ee5c60776 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:29.435806 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1325/2667] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:53:29 2022] POST /identity/v3/users => generated 288 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:53:29.435806 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98d980d5-9cbf-4f7d-8910-d574b285eb07 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:29.436214 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aaf022e-306e-4003-83f1-0f1ee5c60776 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:29.438329 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-98d980d5-9cbf-4f7d-8910-d574b285eb07 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:29.443027 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485d135d-9919-4899-877d-8081bf396738 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:29.444126 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485d135d-9919-4899-877d-8081bf396738 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:29.444637 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485d135d-9919-4899-877d-8081bf396738 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:29.445192 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485d135d-9919-4899-877d-8081bf396738 admin admin] PATH_INFO: `/v3/users/39a6689ef920461982651b13d4cbc65c` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:29.447410 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-485d135d-9919-4899-877d-8081bf396738 admin admin] RBAC: Authorizing `identity:delete_user(user_id=39a6689ef920461982651b13d4cbc65c)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:29.451763 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-485d135d-9919-4899-877d-8081bf396738 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:29.452838 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98d980d5-9cbf-4f7d-8910-d574b285eb07 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:29.458163 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-687e3e13-e4ac-4840-9d28-44e22013c3a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:29.458696 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687e3e13-e4ac-4840-9d28-44e22013c3a1 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:29.458976 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687e3e13-e4ac-4840-9d28-44e22013c3a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:29.459311 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687e3e13-e4ac-4840-9d28-44e22013c3a1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:29.460025 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-687e3e13-e4ac-4840-9d28-44e22013c3a1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:29.461808 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-687e3e13-e4ac-4840-9d28-44e22013c3a1 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:29.475771 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1326/2668] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:53:29 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:53:29.476826 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-687e3e13-e4ac-4840-9d28-44e22013c3a1 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:29.491583 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-687e3e13-e4ac-4840-9d28-44e22013c3a1 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:29.498535 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a7ce01-08c4-458d-a9fb-3ecb095ce8e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:29.499175 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a7ce01-08c4-458d-a9fb-3ecb095ce8e4 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:29.499449 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a7ce01-08c4-458d-a9fb-3ecb095ce8e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:29.499823 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a7ce01-08c4-458d-a9fb-3ecb095ce8e4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:29.500384 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55a7ce01-08c4-458d-a9fb-3ecb095ce8e4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:29.502325 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55a7ce01-08c4-458d-a9fb-3ecb095ce8e4 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:29.517579 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1327/2669] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:53:29 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:53:29.519131 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a7ce01-08c4-458d-a9fb-3ecb095ce8e4 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:29.525688 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:29.533774 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a7ce01-08c4-458d-a9fb-3ecb095ce8e4 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:29.540954 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-485d135d-9919-4899-877d-8081bf396738 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '39a6689ef920461982651b13d4cbc65c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:29.547376 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-053dfbdb-ca6f-41d7-bc7b-170d18ac0072 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:29.548611 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053dfbdb-ca6f-41d7-bc7b-170d18ac0072 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:29.549206 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053dfbdb-ca6f-41d7-bc7b-170d18ac0072 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:29.549644 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053dfbdb-ca6f-41d7-bc7b-170d18ac0072 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:29.550873 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-053dfbdb-ca6f-41d7-bc7b-170d18ac0072 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:29.558586 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-485d135d-9919-4899-877d-8081bf396738 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '39a6689ef920461982651b13d4cbc65c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:29.558844 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-053dfbdb-ca6f-41d7-bc7b-170d18ac0072 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:29.558976 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-485d135d-9919-4899-877d-8081bf396738 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '39a6689ef920461982651b13d4cbc65c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:29.558976 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-485d135d-9919-4899-877d-8081bf396738 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '39a6689ef920461982651b13d4cbc65c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:29.568713 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-485d135d-9919-4899-877d-8081bf396738 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '39a6689ef920461982651b13d4cbc65c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:29.579540 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1328/2670] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:53:29 2022] GET /identity/v3/roles => generated 1420 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:53:29.581973 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-053dfbdb-ca6f-41d7-bc7b-170d18ac0072 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:29.585358 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-053dfbdb-ca6f-41d7-bc7b-170d18ac0072 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:29.600805 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-053dfbdb-ca6f-41d7-bc7b-170d18ac0072 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:29.601397 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1343/2671] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:53:29 2022] DELETE /identity/v3/users/39a6689ef920461982651b13d4cbc65c => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:29.603066 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485d135d-9919-4899-877d-8081bf396738 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:29.605454 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-485d135d-9919-4899-877d-8081bf396738 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:29.637724 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=85,keystone:INSERT=5,keystone:DELETE=10 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:29.720986 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64840a8-5a1e-4631-941f-f76b1840ca7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:29.721941 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64840a8-5a1e-4631-941f-f76b1840ca7b admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:29.721941 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64840a8-5a1e-4631-941f-f76b1840ca7b admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:29.722295 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64840a8-5a1e-4631-941f-f76b1840ca7b admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:29.723476 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64840a8-5a1e-4631-941f-f76b1840ca7b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:29.724864 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e64840a8-5a1e-4631-941f-f76b1840ca7b admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:29.740652 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485d135d-9919-4899-877d-8081bf396738 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:29.746670 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1329/2672] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:53:29 2022] GET /identity/v3/roles => generated 1420 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:53:29.748890 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64840a8-5a1e-4631-941f-f76b1840ca7b admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:29.754663 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89eace31-39b3-49f5-b9a2-ff318d5c38d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:29.755943 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89eace31-39b3-49f5-b9a2-ff318d5c38d3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:29.756651 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89eace31-39b3-49f5-b9a2-ff318d5c38d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:29.757382 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89eace31-39b3-49f5-b9a2-ff318d5c38d3 admin admin] PATH_INFO: `/v3/projects/35b80046f04941e986abdd9298850e08/users/6428001375974a64b4e3d09ef0cac21c/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:29.765741 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64840a8-5a1e-4631-941f-f76b1840ca7b admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:29.773931 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6711bc8-47bb-4437-8cc0-87d9257cc514 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:29.774964 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6711bc8-47bb-4437-8cc0-87d9257cc514 admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:29.775293 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6711bc8-47bb-4437-8cc0-87d9257cc514 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:29.775755 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6711bc8-47bb-4437-8cc0-87d9257cc514 admin admin] PATH_INFO: `/v3/projects/8fd998b657cf4b9c92fdbd73f0d7acce/users/1010cf91e3d14a74925444d66ae47663/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:29.791002 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89eace31-39b3-49f5-b9a2-ff318d5c38d3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=35b80046f04941e986abdd9298850e08, user_id=6428001375974a64b4e3d09ef0cac21c, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:29.798027 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89eace31-39b3-49f5-b9a2-ff318d5c38d3 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:29.805407 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6711bc8-47bb-4437-8cc0-87d9257cc514 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8fd998b657cf4b9c92fdbd73f0d7acce, user_id=1010cf91e3d14a74925444d66ae47663, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:29.812396 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6711bc8-47bb-4437-8cc0-87d9257cc514 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:29.826294 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1344/2673] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:53:29 2022] PUT /identity/v3/projects/35b80046f04941e986abdd9298850e08/users/6428001375974a64b4e3d09ef0cac21c/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:29.836468 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1330/2674] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:53:29 2022] PUT /identity/v3/projects/8fd998b657cf4b9c92fdbd73f0d7acce/users/1010cf91e3d14a74925444d66ae47663/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 89 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:29.850680 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f967c1be-943c-4819-93ab-f1ab3953a6a1 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:29.850680 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f967c1be-943c-4819-93ab-f1ab3953a6a1 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:29.850680 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f967c1be-943c-4819-93ab-f1ab3953a6a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:29.855184 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39cda5d-9598-41fa-a2c2-33aa2c6cbc63 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:29.855527 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39cda5d-9598-41fa-a2c2-33aa2c6cbc63 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:29.855752 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39cda5d-9598-41fa-a2c2-33aa2c6cbc63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:29.892591 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-f967c1be-943c-4819-93ab-f1ab3953a6a1 None None] MFA Rules not processed for user `6428001375974a64b4e3d09ef0cac21c`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:29.893992 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-f967c1be-943c-4819-93ab-f1ab3953a6a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:29.894211 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-a39cda5d-9598-41fa-a2c2-33aa2c6cbc63 None None] MFA Rules not processed for user `1010cf91e3d14a74925444d66ae47663`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:29.895360 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-a39cda5d-9598-41fa-a2c2-33aa2c6cbc63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:29.986615 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1345/2675] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:29 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:53:29.987490 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f967c1be-943c-4819-93ab-f1ab3953a6a1 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:29.988922 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-f967c1be-943c-4819-93ab-f1ab3953a6a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:30.012700 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1331/2676] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:29 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:53:30.012700 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a39cda5d-9598-41fa-a2c2-33aa2c6cbc63 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:30.020162 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-a39cda5d-9598-41fa-a2c2-33aa2c6cbc63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:30.086130 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f967c1be-943c-4819-93ab-f1ab3953a6a1 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:30.091967 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8955a8c3-cd3f-4c34-ba49-bdb52fedc383 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:30.092739 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8955a8c3-cd3f-4c34-ba49-bdb52fedc383 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:30.093104 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8955a8c3-cd3f-4c34-ba49-bdb52fedc383 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:30.093338 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a39cda5d-9598-41fa-a2c2-33aa2c6cbc63 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:30.093680 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8955a8c3-cd3f-4c34-ba49-bdb52fedc383 admin admin] PATH_INFO: `/v3/projects/44b475d014c44b189041dda9b761576e` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:30.094653 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8955a8c3-cd3f-4c34-ba49-bdb52fedc383 admin admin] RBAC: Authorizing `identity:delete_project(project_id=44b475d014c44b189041dda9b761576e)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:30.096704 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8955a8c3-cd3f-4c34-ba49-bdb52fedc383 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:30.099010 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d91e2c-2a67-42cd-8a2d-a4a2359fffac service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:30.099987 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d91e2c-2a67-42cd-8a2d-a4a2359fffac service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:30.100440 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d91e2c-2a67-42cd-8a2d-a4a2359fffac service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:30.100819 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d91e2c-2a67-42cd-8a2d-a4a2359fffac service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:30.111702 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:30.115645 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d91e2c-2a67-42cd-8a2d-a4a2359fffac service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:30.117170 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d91e2c-2a67-42cd-8a2d-a4a2359fffac service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:30.134477 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1332/2677] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:53:30 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:53:30.136251 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d91e2c-2a67-42cd-8a2d-a4a2359fffac service cinder] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:30.138328 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-91d91e2c-2a67-42cd-8a2d-a4a2359fffac service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:30.178331 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=1,keystone:DELETE=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:30.198826 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-8955a8c3-cd3f-4c34-ba49-bdb52fedc383 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '44b475d014c44b189041dda9b761576e', 'request_id': 'req-8955a8c3-cd3f-4c34-ba49-bdb52fedc383'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:30.207314 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-8955a8c3-cd3f-4c34-ba49-bdb52fedc383 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '44b475d014c44b189041dda9b761576e', 'request_id': 'req-8955a8c3-cd3f-4c34-ba49-bdb52fedc383'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:30.217194 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-8955a8c3-cd3f-4c34-ba49-bdb52fedc383 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '44b475d014c44b189041dda9b761576e', 'request_id': 'req-8955a8c3-cd3f-4c34-ba49-bdb52fedc383'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:30.230702 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-8955a8c3-cd3f-4c34-ba49-bdb52fedc383 admin admin] The token cache is being invalidate because project 44b475d014c44b189041dda9b761576e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:53:30.231222 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-8955a8c3-cd3f-4c34-ba49-bdb52fedc383 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:30.232512 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-8955a8c3-cd3f-4c34-ba49-bdb52fedc383 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:30.234518 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1346/2678] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:53:29 2022] DELETE /identity/v3/projects/44b475d014c44b189041dda9b761576e => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:30.239570 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20f5faf-7bf5-4235-981b-4985db9b101a None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:30.240029 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20f5faf-7bf5-4235-981b-4985db9b101a None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:30.240295 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20f5faf-7bf5-4235-981b-4985db9b101a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:30.251665 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d91e2c-2a67-42cd-8a2d-a4a2359fffac service cinder] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:30.263970 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-c20f5faf-7bf5-4235-981b-4985db9b101a None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:30.265344 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-c20f5faf-7bf5-4235-981b-4985db9b101a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:30.336548 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa5ea10e-cdf0-4888-a52e-2445531f4b66 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f4b2ea9aa31445d69e0c1d96b5657708', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:30.337480 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa5ea10e-cdf0-4888-a52e-2445531f4b66 service glance] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:30.337821 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa5ea10e-cdf0-4888-a52e-2445531f4b66 service glance] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:30.338148 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa5ea10e-cdf0-4888-a52e-2445531f4b66 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:30.340455 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-fa5ea10e-cdf0-4888-a52e-2445531f4b66 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:30.360962 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1347/2679] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:53:30.375109 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a17419-dd3f-43ae-94ea-ef240c38d362 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:30.376247 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a17419-dd3f-43ae-94ea-ef240c38d362 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:30.376247 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a17419-dd3f-43ae-94ea-ef240c38d362 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:30.407467 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-34a17419-dd3f-43ae-94ea-ef240c38d362 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:30.408738 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-34a17419-dd3f-43ae-94ea-ef240c38d362 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:30.418186 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1348/2680] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:53:30.423947 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a17419-dd3f-43ae-94ea-ef240c38d362 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:30.433462 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa5ea10e-cdf0-4888-a52e-2445531f4b66 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:30.435874 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a17419-dd3f-43ae-94ea-ef240c38d362 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:30.436072 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa5ea10e-cdf0-4888-a52e-2445531f4b66 service glance] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:30.441357 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cde9db8-dafe-4f56-b275-4a12e1947440 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:30.442005 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cde9db8-dafe-4f56-b275-4a12e1947440 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:30.442288 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cde9db8-dafe-4f56-b275-4a12e1947440 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:30.444098 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cde9db8-dafe-4f56-b275-4a12e1947440 admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:30.444817 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cde9db8-dafe-4f56-b275-4a12e1947440 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:53:30.445136 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cde9db8-dafe-4f56-b275-4a12e1947440 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:30.447034 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cde9db8-dafe-4f56-b275-4a12e1947440 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:30.450149 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1333/2681] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:53:30 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:53:30.468684 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1349/2682] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:53:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:53:30.473384 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa5ea10e-cdf0-4888-a52e-2445531f4b66 service glance] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:30.486214 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa5ea10e-cdf0-4888-a52e-2445531f4b66 service glance] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:30.492939 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356107a0-2755-4d80-a75f-3b609f652bfa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:30.493596 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356107a0-2755-4d80-a75f-3b609f652bfa admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:30.493874 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356107a0-2755-4d80-a75f-3b609f652bfa admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:30.494146 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356107a0-2755-4d80-a75f-3b609f652bfa admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:30.494827 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356107a0-2755-4d80-a75f-3b609f652bfa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:30.496817 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356107a0-2755-4d80-a75f-3b609f652bfa admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:30.543838 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1334/2683] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:53:30 2022] POST /identity/v3/projects => generated 385 bytes in 71 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:53:30.551319 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cde9db8-dafe-4f56-b275-4a12e1947440 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:30.573376 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cde9db8-dafe-4f56-b275-4a12e1947440 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:30.677128 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd66654-300b-4ede-a4fc-bf81b6c33825 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:30.678256 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd66654-300b-4ede-a4fc-bf81b6c33825 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:30.678899 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd66654-300b-4ede-a4fc-bf81b6c33825 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:30.679673 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd66654-300b-4ede-a4fc-bf81b6c33825 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:30.680865 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd66654-300b-4ede-a4fc-bf81b6c33825 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:30.685426 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd66654-300b-4ede-a4fc-bf81b6c33825 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:30.722883 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1350/2684] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:53:30 2022] POST /identity/v3/users => generated 293 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:53:30.728874 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356107a0-2755-4d80-a75f-3b609f652bfa admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:30.748288 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356107a0-2755-4d80-a75f-3b609f652bfa admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:30.754456 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a1970bc-6d9f-453a-8fa9-e5f5b00e3f7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:30.758044 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a1970bc-6d9f-453a-8fa9-e5f5b00e3f7e admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:30.758044 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a1970bc-6d9f-453a-8fa9-e5f5b00e3f7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:30.758044 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a1970bc-6d9f-453a-8fa9-e5f5b00e3f7e admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:30.758044 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a1970bc-6d9f-453a-8fa9-e5f5b00e3f7e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:30.759082 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a1970bc-6d9f-453a-8fa9-e5f5b00e3f7e admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:30.777117 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1335/2685] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:53:30 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:53:30.781994 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd66654-300b-4ede-a4fc-bf81b6c33825 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:30.792743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd66654-300b-4ede-a4fc-bf81b6c33825 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:30.798734 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5cbc635-3c5d-4b13-a0a0-455e6acc2f8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:30.799365 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cbc635-3c5d-4b13-a0a0-455e6acc2f8e admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:30.799649 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cbc635-3c5d-4b13-a0a0-455e6acc2f8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:30.799928 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cbc635-3c5d-4b13-a0a0-455e6acc2f8e admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:30.800615 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5cbc635-3c5d-4b13-a0a0-455e6acc2f8e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:30.802429 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5cbc635-3c5d-4b13-a0a0-455e6acc2f8e admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:30.814229 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1351/2686] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:53:30 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:53:30.818866 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a1970bc-6d9f-453a-8fa9-e5f5b00e3f7e admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:30.830273 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a1970bc-6d9f-453a-8fa9-e5f5b00e3f7e admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:30.835893 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-709c1d89-542a-42df-8051-e623556763f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:30.836525 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709c1d89-542a-42df-8051-e623556763f1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:30.836924 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709c1d89-542a-42df-8051-e623556763f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:30.837275 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709c1d89-542a-42df-8051-e623556763f1 admin admin] PATH_INFO: `/v3/projects/d3734f85ac7342ea85b0fd9ddb4e2531/users/b7ab304c203a43ae9109ba50fa151915/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:30.862510 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-709c1d89-542a-42df-8051-e623556763f1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d3734f85ac7342ea85b0fd9ddb4e2531, user_id=b7ab304c203a43ae9109ba50fa151915, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:30.865159 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-709c1d89-542a-42df-8051-e623556763f1 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:30.884541 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1336/2687] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:53:30 2022] PUT /identity/v3/projects/d3734f85ac7342ea85b0fd9ddb4e2531/users/b7ab304c203a43ae9109ba50fa151915/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:30.897055 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4741de-9327-403c-adb0-5c643773c07b None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:30.897361 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4741de-9327-403c-adb0-5c643773c07b None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:30.897674 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4741de-9327-403c-adb0-5c643773c07b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:30.932243 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-9d4741de-9327-403c-adb0-5c643773c07b None None] MFA Rules not processed for user `b7ab304c203a43ae9109ba50fa151915`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:30.934029 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-9d4741de-9327-403c-adb0-5c643773c07b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:30.951390 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:DELETE=5,keystone:SELECT=91,keystone:INSERT=4 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:31.044212 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1352/2688] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:30 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:53:31.048688 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-709c1d89-542a-42df-8051-e623556763f1 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:31.063672 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:31.136147 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-709c1d89-542a-42df-8051-e623556763f1 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:31.143059 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66398618-3795-459c-ac5c-bcb0481dfa11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:31.143901 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66398618-3795-459c-ac5c-bcb0481dfa11 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:31.144348 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66398618-3795-459c-ac5c-bcb0481dfa11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:31.144756 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66398618-3795-459c-ac5c-bcb0481dfa11 admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:31.145650 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66398618-3795-459c-ac5c-bcb0481dfa11 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:31.148378 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66398618-3795-459c-ac5c-bcb0481dfa11 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:31.182046 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1337/2689] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:53:31 2022] POST /identity/v3/projects => generated 383 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:53:31.186854 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4741de-9327-403c-adb0-5c643773c07b None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:31.197835 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4741de-9327-403c-adb0-5c643773c07b None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:31.270870 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4017ade2-efa4-4e80-8cc5-f62bb7b8fc73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:31.271882 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4017ade2-efa4-4e80-8cc5-f62bb7b8fc73 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:31.272256 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4017ade2-efa4-4e80-8cc5-f62bb7b8fc73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:31.272550 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4017ade2-efa4-4e80-8cc5-f62bb7b8fc73 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:31.273204 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4017ade2-efa4-4e80-8cc5-f62bb7b8fc73 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:31.275563 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4017ade2-efa4-4e80-8cc5-f62bb7b8fc73 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:31.304642 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1353/2690] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:53:31 2022] POST /identity/v3/users => generated 298 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:53:31.309580 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66398618-3795-459c-ac5c-bcb0481dfa11 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:31.321442 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66398618-3795-459c-ac5c-bcb0481dfa11 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:31.328423 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d8a8bc-5746-40a6-aad3-5ccf4a6e67b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:31.329617 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d8a8bc-5746-40a6-aad3-5ccf4a6e67b9 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:31.330082 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d8a8bc-5746-40a6-aad3-5ccf4a6e67b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:31.330565 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d8a8bc-5746-40a6-aad3-5ccf4a6e67b9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:31.331668 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64d8a8bc-5746-40a6-aad3-5ccf4a6e67b9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:31.334462 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64d8a8bc-5746-40a6-aad3-5ccf4a6e67b9 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:31.353722 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1338/2691] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:53:31 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:53:31.358893 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4017ade2-efa4-4e80-8cc5-f62bb7b8fc73 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:31.369396 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4017ade2-efa4-4e80-8cc5-f62bb7b8fc73 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:31.374904 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770dddaf-1649-4bb6-b83d-9fbc01f7f4ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:31.375579 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770dddaf-1649-4bb6-b83d-9fbc01f7f4ee admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:31.376163 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770dddaf-1649-4bb6-b83d-9fbc01f7f4ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:31.376163 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770dddaf-1649-4bb6-b83d-9fbc01f7f4ee admin admin] PATH_INFO: `/v3/projects/7bead82d9b3444e49f159b65f5e3a520/users/e5dd5a0f525f4ab78aaa0bd6347a82f1/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:31.393396 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-770dddaf-1649-4bb6-b83d-9fbc01f7f4ee admin admin] RBAC: Authorizing `identity:create_grant(project_id=7bead82d9b3444e49f159b65f5e3a520, user_id=e5dd5a0f525f4ab78aaa0bd6347a82f1, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:31.396610 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-770dddaf-1649-4bb6-b83d-9fbc01f7f4ee admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:31.411015 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1354/2692] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:53:31 2022] PUT /identity/v3/projects/7bead82d9b3444e49f159b65f5e3a520/users/e5dd5a0f525f4ab78aaa0bd6347a82f1/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:53:31.417733 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d8a8bc-5746-40a6-aad3-5ccf4a6e67b9 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:31.428966 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d8a8bc-5746-40a6-aad3-5ccf4a6e67b9 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:31.491928 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f2f3a2-d34a-445b-b964-daf677951ab1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:31.492559 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f2f3a2-d34a-445b-b964-daf677951ab1 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:31.492964 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f2f3a2-d34a-445b-b964-daf677951ab1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:31.493135 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f2f3a2-d34a-445b-b964-daf677951ab1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:31.493791 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f2f3a2-d34a-445b-b964-daf677951ab1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:31.495455 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f2f3a2-d34a-445b-b964-daf677951ab1 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:31.507107 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1339/2693] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:53:31 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:53:31.511806 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770dddaf-1649-4bb6-b83d-9fbc01f7f4ee admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:31.524894 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770dddaf-1649-4bb6-b83d-9fbc01f7f4ee admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:31.529965 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a81d30-438d-46d1-9c53-51c52016c990 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:31.531218 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a81d30-438d-46d1-9c53-51c52016c990 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:31.531218 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a81d30-438d-46d1-9c53-51c52016c990 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:31.531466 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a81d30-438d-46d1-9c53-51c52016c990 admin admin] PATH_INFO: `/v3/projects/7bead82d9b3444e49f159b65f5e3a520/users/e5dd5a0f525f4ab78aaa0bd6347a82f1/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:31.533958 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a81d30-438d-46d1-9c53-51c52016c990 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7bead82d9b3444e49f159b65f5e3a520, user_id=e5dd5a0f525f4ab78aaa0bd6347a82f1, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:31.537347 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a81d30-438d-46d1-9c53-51c52016c990 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:31.551704 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1355/2694] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:53:31 2022] PUT /identity/v3/projects/7bead82d9b3444e49f159b65f5e3a520/users/e5dd5a0f525f4ab78aaa0bd6347a82f1/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:31.569100 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3308143-4b02-4e2f-9115-68ec1f9337c1 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:31.569612 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3308143-4b02-4e2f-9115-68ec1f9337c1 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:31.570135 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3308143-4b02-4e2f-9115-68ec1f9337c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:31.610005 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-c3308143-4b02-4e2f-9115-68ec1f9337c1 None None] MFA Rules not processed for user `e5dd5a0f525f4ab78aaa0bd6347a82f1`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:31.611294 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-c3308143-4b02-4e2f-9115-68ec1f9337c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:31.728338 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1340/2695] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:31 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:53:31.741630 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a81d30-438d-46d1-9c53-51c52016c990 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:31.744400 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-c5a81d30-438d-46d1-9c53-51c52016c990 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:31.837287 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a81d30-438d-46d1-9c53-51c52016c990 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:31.841987 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9efeec83-3a1d-4670-bc2c-f06e9ae10019 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:31.842836 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9efeec83-3a1d-4670-bc2c-f06e9ae10019 service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:31.843223 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9efeec83-3a1d-4670-bc2c-f06e9ae10019 service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:31.843384 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9efeec83-3a1d-4670-bc2c-f06e9ae10019 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:31.864469 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:31.929803 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9efeec83-3a1d-4670-bc2c-f06e9ae10019 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:31.931440 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9efeec83-3a1d-4670-bc2c-f06e9ae10019 service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:31.944326 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1356/2696] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:53:31 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 203 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:53:36.717238 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3308143-4b02-4e2f-9115-68ec1f9337c1 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:36.718020 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-c3308143-4b02-4e2f-9115-68ec1f9337c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:36.726440 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:36.731384 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3308143-4b02-4e2f-9115-68ec1f9337c1 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:36.743957 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa7edee6-5720-4008-8591-6738d57a1b91 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:36.744924 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7edee6-5720-4008-8591-6738d57a1b91 service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:36.745618 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7edee6-5720-4008-8591-6738d57a1b91 service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:36.746169 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa7edee6-5720-4008-8591-6738d57a1b91 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:36.763935 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa7edee6-5720-4008-8591-6738d57a1b91 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:36.765634 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa7edee6-5720-4008-8591-6738d57a1b91 service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:36.776976 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1341/2697] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:53:36 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 61 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:53:39.510080 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9efeec83-3a1d-4670-bc2c-f06e9ae10019 service cinder] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:39.511933 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-9efeec83-3a1d-4670-bc2c-f06e9ae10019 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:39.600095 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9efeec83-3a1d-4670-bc2c-f06e9ae10019 service cinder] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:39.608057 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e360d6-4458-480a-acc7-ee351c24ea6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:39.609073 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e360d6-4458-480a-acc7-ee351c24ea6b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:39.609453 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e360d6-4458-480a-acc7-ee351c24ea6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:39.609928 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e360d6-4458-480a-acc7-ee351c24ea6b admin admin] PATH_INFO: `/v3/users/6428001375974a64b4e3d09ef0cac21c` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:39.611630 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e360d6-4458-480a-acc7-ee351c24ea6b admin admin] RBAC: Authorizing `identity:delete_user(user_id=6428001375974a64b4e3d09ef0cac21c)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:39.613683 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0e360d6-4458-480a-acc7-ee351c24ea6b admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:39.688175 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-d0e360d6-4458-480a-acc7-ee351c24ea6b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6428001375974a64b4e3d09ef0cac21c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:39.698222 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-d0e360d6-4458-480a-acc7-ee351c24ea6b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6428001375974a64b4e3d09ef0cac21c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:39.702493 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-d0e360d6-4458-480a-acc7-ee351c24ea6b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6428001375974a64b4e3d09ef0cac21c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:39.702493 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-d0e360d6-4458-480a-acc7-ee351c24ea6b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6428001375974a64b4e3d09ef0cac21c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:39.715237 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-d0e360d6-4458-480a-acc7-ee351c24ea6b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6428001375974a64b4e3d09ef0cac21c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:39.742234 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1357/2698] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:53:39 2022] DELETE /identity/v3/users/6428001375974a64b4e3d09ef0cac21c => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:53:39.747686 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3ad51e-c2bb-4019-b264-610e93d85ac7 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:39.748424 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3ad51e-c2bb-4019-b264-610e93d85ac7 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:39.748424 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3ad51e-c2bb-4019-b264-610e93d85ac7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:39.771310 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-6b3ad51e-c2bb-4019-b264-610e93d85ac7 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:39.772583 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-6b3ad51e-c2bb-4019-b264-610e93d85ac7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:39.879824 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1342/2699] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:53:39.893150 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e360d6-4458-480a-acc7-ee351c24ea6b admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:39.896258 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d0e360d6-4458-480a-acc7-ee351c24ea6b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:39.952366 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:39.972407 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0e360d6-4458-480a-acc7-ee351c24ea6b admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:39.976192 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ecd0d0d-c6f5-4ac5-a1f0-6be93decfb08 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:39.976850 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ecd0d0d-c6f5-4ac5-a1f0-6be93decfb08 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:39.977127 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ecd0d0d-c6f5-4ac5-a1f0-6be93decfb08 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:39.977395 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ecd0d0d-c6f5-4ac5-a1f0-6be93decfb08 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:39.988359 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ecd0d0d-c6f5-4ac5-a1f0-6be93decfb08 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:39.989894 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ecd0d0d-c6f5-4ac5-a1f0-6be93decfb08 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:40.001682 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1358/2700] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:53:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:53:40.331727 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3ad51e-c2bb-4019-b264-610e93d85ac7 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:40.333368 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-6b3ad51e-c2bb-4019-b264-610e93d85ac7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:40.356018 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3ad51e-c2bb-4019-b264-610e93d85ac7 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:40.362839 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fabae031-cfd2-4708-8e13-7d57e925b006 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:40.363657 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fabae031-cfd2-4708-8e13-7d57e925b006 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:40.363912 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fabae031-cfd2-4708-8e13-7d57e925b006 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:40.364460 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fabae031-cfd2-4708-8e13-7d57e925b006 admin admin] PATH_INFO: `/v3/projects/35b80046f04941e986abdd9298850e08` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:40.366118 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fabae031-cfd2-4708-8e13-7d57e925b006 admin admin] RBAC: Authorizing `identity:delete_project(project_id=35b80046f04941e986abdd9298850e08)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:40.368437 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fabae031-cfd2-4708-8e13-7d57e925b006 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:40.421638 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-fabae031-cfd2-4708-8e13-7d57e925b006 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '35b80046f04941e986abdd9298850e08', 'request_id': 'req-fabae031-cfd2-4708-8e13-7d57e925b006'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:40.428734 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-fabae031-cfd2-4708-8e13-7d57e925b006 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '35b80046f04941e986abdd9298850e08', 'request_id': 'req-fabae031-cfd2-4708-8e13-7d57e925b006'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:40.438125 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-fabae031-cfd2-4708-8e13-7d57e925b006 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '35b80046f04941e986abdd9298850e08', 'request_id': 'req-fabae031-cfd2-4708-8e13-7d57e925b006'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:40.453644 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-fabae031-cfd2-4708-8e13-7d57e925b006 admin admin] The token cache is being invalidate because project 35b80046f04941e986abdd9298850e08 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:53:40.453942 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-fabae031-cfd2-4708-8e13-7d57e925b006 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:40.454713 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-fabae031-cfd2-4708-8e13-7d57e925b006 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:40.456502 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1343/2701] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:53:40 2022] DELETE /identity/v3/projects/35b80046f04941e986abdd9298850e08 => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:40.462070 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88cd71e1-7c4a-4847-a591-0282f4416ba7 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:40.462340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88cd71e1-7c4a-4847-a591-0282f4416ba7 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:40.462797 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88cd71e1-7c4a-4847-a591-0282f4416ba7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:40.488317 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-88cd71e1-7c4a-4847-a591-0282f4416ba7 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:40.489569 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-88cd71e1-7c4a-4847-a591-0282f4416ba7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:40.562004 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1359/2702] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:53:40.568739 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e958700-ea25-4f57-8851-1cf0e96db97d None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:40.569103 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e958700-ea25-4f57-8851-1cf0e96db97d None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:40.569300 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e958700-ea25-4f57-8851-1cf0e96db97d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:40.592916 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-9e958700-ea25-4f57-8851-1cf0e96db97d None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:40.594483 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-9e958700-ea25-4f57-8851-1cf0e96db97d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:40.604065 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1344/2703] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:53:40.609596 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88cd71e1-7c4a-4847-a591-0282f4416ba7 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:40.620286 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88cd71e1-7c4a-4847-a591-0282f4416ba7 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:40.625110 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54979643-e093-45df-b448-c4dc2f127dd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:40.625747 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54979643-e093-45df-b448-c4dc2f127dd9 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:40.626024 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54979643-e093-45df-b448-c4dc2f127dd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:40.626324 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54979643-e093-45df-b448-c4dc2f127dd9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:40.627026 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54979643-e093-45df-b448-c4dc2f127dd9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:53:40.627336 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54979643-e093-45df-b448-c4dc2f127dd9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:40.628967 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54979643-e093-45df-b448-c4dc2f127dd9 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:40.642447 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1360/2704] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:53:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:53:40.647325 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e958700-ea25-4f57-8851-1cf0e96db97d None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:40.658171 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e958700-ea25-4f57-8851-1cf0e96db97d None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:40.663630 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497ef916-c4e1-4b43-8865-0d6c7f9fa558 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:40.664261 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497ef916-c4e1-4b43-8865-0d6c7f9fa558 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:40.664546 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497ef916-c4e1-4b43-8865-0d6c7f9fa558 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:40.664823 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497ef916-c4e1-4b43-8865-0d6c7f9fa558 admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:40.665478 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-497ef916-c4e1-4b43-8865-0d6c7f9fa558 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:40.667480 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-497ef916-c4e1-4b43-8865-0d6c7f9fa558 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:40.701767 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1345/2705] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:53:40 2022] POST /identity/v3/projects => generated 381 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:53:40.706227 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54979643-e093-45df-b448-c4dc2f127dd9 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:40.717723 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54979643-e093-45df-b448-c4dc2f127dd9 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:40.793533 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6184ca6-91bc-47d4-b2cb-5f9e77887d77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:40.794938 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6184ca6-91bc-47d4-b2cb-5f9e77887d77 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:40.795362 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6184ca6-91bc-47d4-b2cb-5f9e77887d77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:40.795610 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6184ca6-91bc-47d4-b2cb-5f9e77887d77 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:40.796439 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6184ca6-91bc-47d4-b2cb-5f9e77887d77 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:40.798819 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6184ca6-91bc-47d4-b2cb-5f9e77887d77 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:40.824263 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1361/2706] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:53:40 2022] POST /identity/v3/users => generated 291 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:53:40.829968 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497ef916-c4e1-4b43-8865-0d6c7f9fa558 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:40.843899 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497ef916-c4e1-4b43-8865-0d6c7f9fa558 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:40.853489 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5c9b27-7f6e-4e13-ac6c-dd7469f33e18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:40.854641 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5c9b27-7f6e-4e13-ac6c-dd7469f33e18 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:40.855145 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5c9b27-7f6e-4e13-ac6c-dd7469f33e18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:40.855838 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5c9b27-7f6e-4e13-ac6c-dd7469f33e18 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:40.857425 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e5c9b27-7f6e-4e13-ac6c-dd7469f33e18 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:40.859243 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e5c9b27-7f6e-4e13-ac6c-dd7469f33e18 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:40.872623 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1346/2707] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:53:40 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:53:40.876223 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6184ca6-91bc-47d4-b2cb-5f9e77887d77 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:40.887740 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6184ca6-91bc-47d4-b2cb-5f9e77887d77 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:40.893128 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4e4b19-fdfc-4624-91d0-863918463279 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:40.893760 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4e4b19-fdfc-4624-91d0-863918463279 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:40.894035 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4e4b19-fdfc-4624-91d0-863918463279 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:40.894304 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4e4b19-fdfc-4624-91d0-863918463279 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:40.895001 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a4e4b19-fdfc-4624-91d0-863918463279 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:40.896667 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a4e4b19-fdfc-4624-91d0-863918463279 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:40.909493 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1362/2708] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:53:40 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:53:40.914535 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5c9b27-7f6e-4e13-ac6c-dd7469f33e18 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:40.926647 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e5c9b27-7f6e-4e13-ac6c-dd7469f33e18 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:40.931503 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8149d19b-82bb-4898-b220-91260de7ef0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:40.932178 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8149d19b-82bb-4898-b220-91260de7ef0f admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:40.932456 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8149d19b-82bb-4898-b220-91260de7ef0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:40.932730 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8149d19b-82bb-4898-b220-91260de7ef0f admin admin] PATH_INFO: `/v3/projects/c724ddf152e44905b313c1976a4f1574/users/f4c463cced614be791c68b47311bcc23/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:40.959095 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8149d19b-82bb-4898-b220-91260de7ef0f admin admin] RBAC: Authorizing `identity:create_grant(project_id=c724ddf152e44905b313c1976a4f1574, user_id=f4c463cced614be791c68b47311bcc23, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:40.965262 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8149d19b-82bb-4898-b220-91260de7ef0f admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:40.988375 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1347/2709] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:53:40 2022] PUT /identity/v3/projects/c724ddf152e44905b313c1976a4f1574/users/f4c463cced614be791c68b47311bcc23/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 75 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:41.004130 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb7c3d3-3c2f-422c-9685-5ec932ff089a None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:41.004678 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb7c3d3-3c2f-422c-9685-5ec932ff089a None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:41.005005 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb7c3d3-3c2f-422c-9685-5ec932ff089a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:41.049744 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-4eb7c3d3-3c2f-422c-9685-5ec932ff089a None None] MFA Rules not processed for user `f4c463cced614be791c68b47311bcc23`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:41.051273 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4eb7c3d3-3c2f-422c-9685-5ec932ff089a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:41.100464 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:41.141971 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1363/2710] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:41 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:53:41.146932 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8149d19b-82bb-4898-b220-91260de7ef0f admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:41.162194 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:41.225575 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8149d19b-82bb-4898-b220-91260de7ef0f admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:41.230810 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37933e71-1b3a-4dd5-972f-9c04821fa9a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:41.231476 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37933e71-1b3a-4dd5-972f-9c04821fa9a9 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:41.231789 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37933e71-1b3a-4dd5-972f-9c04821fa9a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:41.232066 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37933e71-1b3a-4dd5-972f-9c04821fa9a9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:41.232767 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37933e71-1b3a-4dd5-972f-9c04821fa9a9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:41.234886 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37933e71-1b3a-4dd5-972f-9c04821fa9a9 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:41.267582 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1348/2711] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:53:41 2022] POST /identity/v3/projects => generated 379 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:53:41.272439 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb7c3d3-3c2f-422c-9685-5ec932ff089a None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:41.284898 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb7c3d3-3c2f-422c-9685-5ec932ff089a None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:41.356122 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-888a37b4-3fbd-474b-b6df-75e030ca6399 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:41.357032 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888a37b4-3fbd-474b-b6df-75e030ca6399 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:41.357418 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888a37b4-3fbd-474b-b6df-75e030ca6399 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:41.357946 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888a37b4-3fbd-474b-b6df-75e030ca6399 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:41.359296 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-888a37b4-3fbd-474b-b6df-75e030ca6399 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:41.362320 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-888a37b4-3fbd-474b-b6df-75e030ca6399 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:41.391325 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1364/2712] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:53:41 2022] POST /identity/v3/users => generated 296 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:53:41.395509 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37933e71-1b3a-4dd5-972f-9c04821fa9a9 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:41.412412 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37933e71-1b3a-4dd5-972f-9c04821fa9a9 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:41.421525 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a231f510-0bef-447c-b181-4c39c35e4263 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:41.422864 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a231f510-0bef-447c-b181-4c39c35e4263 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:41.423364 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a231f510-0bef-447c-b181-4c39c35e4263 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:41.423974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a231f510-0bef-447c-b181-4c39c35e4263 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:41.425060 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a231f510-0bef-447c-b181-4c39c35e4263 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:41.428333 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a231f510-0bef-447c-b181-4c39c35e4263 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:41.449707 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1349/2713] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:53:41 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:53:41.455156 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-888a37b4-3fbd-474b-b6df-75e030ca6399 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:41.467747 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-888a37b4-3fbd-474b-b6df-75e030ca6399 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:41.473081 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b74e4da-78f1-40a4-b007-134eaf751a48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:41.473790 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b74e4da-78f1-40a4-b007-134eaf751a48 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:41.474071 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b74e4da-78f1-40a4-b007-134eaf751a48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:41.474349 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b74e4da-78f1-40a4-b007-134eaf751a48 admin admin] PATH_INFO: `/v3/projects/bb30290b9a8a467a81ef70f6949da18b/users/cbc121319cce43ff897a367465693897/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:41.491532 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b74e4da-78f1-40a4-b007-134eaf751a48 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bb30290b9a8a467a81ef70f6949da18b, user_id=cbc121319cce43ff897a367465693897, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:41.494310 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b74e4da-78f1-40a4-b007-134eaf751a48 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:41.514201 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1365/2714] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:53:41 2022] PUT /identity/v3/projects/bb30290b9a8a467a81ef70f6949da18b/users/cbc121319cce43ff897a367465693897/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:53:41.519472 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a231f510-0bef-447c-b181-4c39c35e4263 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:41.533320 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a231f510-0bef-447c-b181-4c39c35e4263 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:41.601873 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ccc6ffc-05d3-4e28-b11f-8831252bcb0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:41.602548 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ccc6ffc-05d3-4e28-b11f-8831252bcb0c admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:41.602867 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ccc6ffc-05d3-4e28-b11f-8831252bcb0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:41.603116 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ccc6ffc-05d3-4e28-b11f-8831252bcb0c admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:41.603775 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ccc6ffc-05d3-4e28-b11f-8831252bcb0c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:41.605412 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ccc6ffc-05d3-4e28-b11f-8831252bcb0c admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:41.625662 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1350/2715] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:53:41 2022] GET /identity/v3/roles => generated 1420 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:53:41.630535 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b74e4da-78f1-40a4-b007-134eaf751a48 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:41.642890 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b74e4da-78f1-40a4-b007-134eaf751a48 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:41.647706 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6901503e-0f45-4885-a1f1-42ec0d90dfbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:41.648383 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6901503e-0f45-4885-a1f1-42ec0d90dfbf admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:41.648672 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6901503e-0f45-4885-a1f1-42ec0d90dfbf admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:41.648949 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6901503e-0f45-4885-a1f1-42ec0d90dfbf admin admin] PATH_INFO: `/v3/projects/bb30290b9a8a467a81ef70f6949da18b/users/cbc121319cce43ff897a367465693897/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:41.651006 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6901503e-0f45-4885-a1f1-42ec0d90dfbf admin admin] RBAC: Authorizing `identity:create_grant(project_id=bb30290b9a8a467a81ef70f6949da18b, user_id=cbc121319cce43ff897a367465693897, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:41.653813 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6901503e-0f45-4885-a1f1-42ec0d90dfbf admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:41.665457 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1366/2716] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:53:41 2022] PUT /identity/v3/projects/bb30290b9a8a467a81ef70f6949da18b/users/cbc121319cce43ff897a367465693897/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:41.676838 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9d2c36-4751-4f45-82ff-f6912f687de0 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:41.677279 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9d2c36-4751-4f45-82ff-f6912f687de0 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:41.677674 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c9d2c36-4751-4f45-82ff-f6912f687de0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:41.709954 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-4c9d2c36-4751-4f45-82ff-f6912f687de0 None None] MFA Rules not processed for user `cbc121319cce43ff897a367465693897`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:41.711284 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-4c9d2c36-4751-4f45-82ff-f6912f687de0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:41.796390 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1351/2717] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:41 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:53:41.809917 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6901503e-0f45-4885-a1f1-42ec0d90dfbf admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:41.812270 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-6901503e-0f45-4885-a1f1-42ec0d90dfbf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:41.900117 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6901503e-0f45-4885-a1f1-42ec0d90dfbf admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:41.906636 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfef537a-3d1d-485e-8ec9-f47ebe8c223c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:41.907743 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfef537a-3d1d-485e-8ec9-f47ebe8c223c service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:41.908263 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfef537a-3d1d-485e-8ec9-f47ebe8c223c service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:41.908762 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfef537a-3d1d-485e-8ec9-f47ebe8c223c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:41.927717 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfef537a-3d1d-485e-8ec9-f47ebe8c223c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:41.930587 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfef537a-3d1d-485e-8ec9-f47ebe8c223c service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:41.950801 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1367/2718] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:53:41 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:53:42.062158 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c9d2c36-4751-4f45-82ff-f6912f687de0 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:42.079514 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c9d2c36-4751-4f45-82ff-f6912f687de0 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:42.084209 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6e44ad-9153-4cb1-bd21-59173e518778 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:42.085012 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6e44ad-9153-4cb1-bd21-59173e518778 service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:42.085355 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6e44ad-9153-4cb1-bd21-59173e518778 service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:42.085669 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6e44ad-9153-4cb1-bd21-59173e518778 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:42.091485 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:42.160566 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfef537a-3d1d-485e-8ec9-f47ebe8c223c service cinder] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:42.164002 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-cfef537a-3d1d-485e-8ec9-f47ebe8c223c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:42.190975 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e6e44ad-9153-4cb1-bd21-59173e518778 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:42.193898 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e6e44ad-9153-4cb1-bd21-59173e518778 service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:42.207716 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:42.209409 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1352/2719] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:53:42 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:53:42.262316 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfef537a-3d1d-485e-8ec9-f47ebe8c223c service cinder] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:42.269238 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a344fe0a-7cc7-4895-b0d8-e2e9f431f141 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:42.270491 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a344fe0a-7cc7-4895-b0d8-e2e9f431f141 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:42.271069 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a344fe0a-7cc7-4895-b0d8-e2e9f431f141 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:42.271586 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a344fe0a-7cc7-4895-b0d8-e2e9f431f141 admin admin] PATH_INFO: `/v3/users/b7ab304c203a43ae9109ba50fa151915` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:42.273799 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a344fe0a-7cc7-4895-b0d8-e2e9f431f141 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b7ab304c203a43ae9109ba50fa151915)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:42.277646 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a344fe0a-7cc7-4895-b0d8-e2e9f431f141 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:42.362221 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a344fe0a-7cc7-4895-b0d8-e2e9f431f141 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b7ab304c203a43ae9109ba50fa151915'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:42.385559 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a344fe0a-7cc7-4895-b0d8-e2e9f431f141 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b7ab304c203a43ae9109ba50fa151915'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:42.388354 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a344fe0a-7cc7-4895-b0d8-e2e9f431f141 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b7ab304c203a43ae9109ba50fa151915'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:42.389288 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a344fe0a-7cc7-4895-b0d8-e2e9f431f141 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b7ab304c203a43ae9109ba50fa151915'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:42.402510 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a344fe0a-7cc7-4895-b0d8-e2e9f431f141 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b7ab304c203a43ae9109ba50fa151915'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:42.429156 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1368/2720] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:53:42 2022] DELETE /identity/v3/users/b7ab304c203a43ae9109ba50fa151915 => generated 0 bytes in 270 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:53:42.434099 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64496c87-b7ee-4448-9aa4-d6786a68f091 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:42.434379 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64496c87-b7ee-4448-9aa4-d6786a68f091 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:42.434812 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64496c87-b7ee-4448-9aa4-d6786a68f091 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:42.456068 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-64496c87-b7ee-4448-9aa4-d6786a68f091 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:42.457353 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-64496c87-b7ee-4448-9aa4-d6786a68f091 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:42.527299 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1353/2721] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:53:42.539583 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a344fe0a-7cc7-4895-b0d8-e2e9f431f141 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:42.541088 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a344fe0a-7cc7-4895-b0d8-e2e9f431f141 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:42.608689 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a344fe0a-7cc7-4895-b0d8-e2e9f431f141 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:42.612761 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11bd9839-5517-41c9-bcde-6881cd844b26 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:42.613390 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bd9839-5517-41c9-bcde-6881cd844b26 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:42.613674 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bd9839-5517-41c9-bcde-6881cd844b26 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:42.613948 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bd9839-5517-41c9-bcde-6881cd844b26 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:42.624525 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11bd9839-5517-41c9-bcde-6881cd844b26 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:42.626205 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11bd9839-5517-41c9-bcde-6881cd844b26 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:42.644019 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1369/2722] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:53:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:53:42.984602 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64496c87-b7ee-4448-9aa4-d6786a68f091 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:42.986158 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-64496c87-b7ee-4448-9aa4-d6786a68f091 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:43.000627 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64496c87-b7ee-4448-9aa4-d6786a68f091 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:43.007325 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653b8fb4-d738-40fb-8df2-b7b69374f240 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:43.008456 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653b8fb4-d738-40fb-8df2-b7b69374f240 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:43.008909 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653b8fb4-d738-40fb-8df2-b7b69374f240 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:43.009488 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653b8fb4-d738-40fb-8df2-b7b69374f240 admin admin] PATH_INFO: `/v3/projects/d3734f85ac7342ea85b0fd9ddb4e2531` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:43.011639 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-653b8fb4-d738-40fb-8df2-b7b69374f240 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d3734f85ac7342ea85b0fd9ddb4e2531)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:43.016295 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-653b8fb4-d738-40fb-8df2-b7b69374f240 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:43.092819 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-653b8fb4-d738-40fb-8df2-b7b69374f240 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd3734f85ac7342ea85b0fd9ddb4e2531', 'request_id': 'req-653b8fb4-d738-40fb-8df2-b7b69374f240'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:43.100215 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-653b8fb4-d738-40fb-8df2-b7b69374f240 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd3734f85ac7342ea85b0fd9ddb4e2531', 'request_id': 'req-653b8fb4-d738-40fb-8df2-b7b69374f240'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:43.109612 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-653b8fb4-d738-40fb-8df2-b7b69374f240 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd3734f85ac7342ea85b0fd9ddb4e2531', 'request_id': 'req-653b8fb4-d738-40fb-8df2-b7b69374f240'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:43.122021 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-653b8fb4-d738-40fb-8df2-b7b69374f240 admin admin] The token cache is being invalidate because project d3734f85ac7342ea85b0fd9ddb4e2531 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:53:43.122891 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-653b8fb4-d738-40fb-8df2-b7b69374f240 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:43.123646 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-653b8fb4-d738-40fb-8df2-b7b69374f240 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:43.125216 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1354/2723] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:53:42 2022] DELETE /identity/v3/projects/d3734f85ac7342ea85b0fd9ddb4e2531 => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:43.129562 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11bd9839-5517-41c9-bcde-6881cd844b26 service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:43.141014 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11bd9839-5517-41c9-bcde-6881cd844b26 service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:43.206796 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:43.219004 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc94949-190e-4e2b-aca8-071a5a10734c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:43.219805 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc94949-190e-4e2b-aca8-071a5a10734c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:43.220095 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc94949-190e-4e2b-aca8-071a5a10734c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:43.220399 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc94949-190e-4e2b-aca8-071a5a10734c admin admin] PATH_INFO: `/v3/users/e5dd5a0f525f4ab78aaa0bd6347a82f1` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:43.221654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc94949-190e-4e2b-aca8-071a5a10734c admin admin] RBAC: Authorizing `identity:delete_user(user_id=e5dd5a0f525f4ab78aaa0bd6347a82f1)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:43.223706 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc94949-190e-4e2b-aca8-071a5a10734c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:43.288323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-5cc94949-190e-4e2b-aca8-071a5a10734c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e5dd5a0f525f4ab78aaa0bd6347a82f1'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:43.297569 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-5cc94949-190e-4e2b-aca8-071a5a10734c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e5dd5a0f525f4ab78aaa0bd6347a82f1'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:43.298248 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-5cc94949-190e-4e2b-aca8-071a5a10734c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e5dd5a0f525f4ab78aaa0bd6347a82f1'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:43.298914 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-5cc94949-190e-4e2b-aca8-071a5a10734c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e5dd5a0f525f4ab78aaa0bd6347a82f1'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:43.307780 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-5cc94949-190e-4e2b-aca8-071a5a10734c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e5dd5a0f525f4ab78aaa0bd6347a82f1'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:43.329104 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1370/2724] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:53:43 2022] DELETE /identity/v3/users/e5dd5a0f525f4ab78aaa0bd6347a82f1 => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:53:43.695902 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653b8fb4-d738-40fb-8df2-b7b69374f240 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:43.701973 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-653b8fb4-d738-40fb-8df2-b7b69374f240 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:43.740763 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653b8fb4-d738-40fb-8df2-b7b69374f240 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:43.776779 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:43.829160 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f5adb1-dfb1-49ad-b6a0-2b775f9475c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:43.830034 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f5adb1-dfb1-49ad-b6a0-2b775f9475c5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:43.830457 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f5adb1-dfb1-49ad-b6a0-2b775f9475c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:43.831993 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f5adb1-dfb1-49ad-b6a0-2b775f9475c5 admin admin] PATH_INFO: `/v3/projects/7bead82d9b3444e49f159b65f5e3a520` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:43.833934 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f5adb1-dfb1-49ad-b6a0-2b775f9475c5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7bead82d9b3444e49f159b65f5e3a520)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:43.837811 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f5adb1-dfb1-49ad-b6a0-2b775f9475c5 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:43.942882 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-b7f5adb1-dfb1-49ad-b6a0-2b775f9475c5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7bead82d9b3444e49f159b65f5e3a520', 'request_id': 'req-b7f5adb1-dfb1-49ad-b6a0-2b775f9475c5'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:43.956882 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-b7f5adb1-dfb1-49ad-b6a0-2b775f9475c5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7bead82d9b3444e49f159b65f5e3a520', 'request_id': 'req-b7f5adb1-dfb1-49ad-b6a0-2b775f9475c5'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:43.981434 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-b7f5adb1-dfb1-49ad-b6a0-2b775f9475c5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7bead82d9b3444e49f159b65f5e3a520', 'request_id': 'req-b7f5adb1-dfb1-49ad-b6a0-2b775f9475c5'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:44.006466 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-b7f5adb1-dfb1-49ad-b6a0-2b775f9475c5 admin admin] The token cache is being invalidate because project 7bead82d9b3444e49f159b65f5e3a520 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:53:44.007035 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-b7f5adb1-dfb1-49ad-b6a0-2b775f9475c5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:44.008210 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-b7f5adb1-dfb1-49ad-b6a0-2b775f9475c5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:44.010770 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1355/2725] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:53:43 2022] DELETE /identity/v3/projects/7bead82d9b3444e49f159b65f5e3a520 => generated 0 bytes in 316 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:44.019695 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b166c8a4-f4fd-4e2f-8644-80598c6f9c54 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:44.020291 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b166c8a4-f4fd-4e2f-8644-80598c6f9c54 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:44.020642 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b166c8a4-f4fd-4e2f-8644-80598c6f9c54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:44.063084 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-b166c8a4-f4fd-4e2f-8644-80598c6f9c54 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:44.064418 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-b166c8a4-f4fd-4e2f-8644-80598c6f9c54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:44.133406 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1371/2726] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:53:44.142879 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eddbae10-508a-4576-9b4b-1d034430a0df None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:44.143774 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eddbae10-508a-4576-9b4b-1d034430a0df None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:44.143774 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eddbae10-508a-4576-9b4b-1d034430a0df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:44.168353 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-eddbae10-508a-4576-9b4b-1d034430a0df None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:44.171156 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-eddbae10-508a-4576-9b4b-1d034430a0df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:44.186290 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1356/2727] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:53:44.190832 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b166c8a4-f4fd-4e2f-8644-80598c6f9c54 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:44.201256 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b166c8a4-f4fd-4e2f-8644-80598c6f9c54 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:44.206285 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-268a9a68-fe46-47df-aacb-d12787ecde3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:44.206938 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268a9a68-fe46-47df-aacb-d12787ecde3b admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:44.207215 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268a9a68-fe46-47df-aacb-d12787ecde3b admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:44.207484 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268a9a68-fe46-47df-aacb-d12787ecde3b admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:44.208139 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-268a9a68-fe46-47df-aacb-d12787ecde3b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:53:44.208438 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-268a9a68-fe46-47df-aacb-d12787ecde3b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:44.210058 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-268a9a68-fe46-47df-aacb-d12787ecde3b admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:44.227005 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1372/2728] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:53:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:53:44.233073 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eddbae10-508a-4576-9b4b-1d034430a0df None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:44.244553 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eddbae10-508a-4576-9b4b-1d034430a0df None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:44.250495 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1143bd7-3bde-43c3-bc26-c0004cf7433a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:44.251124 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1143bd7-3bde-43c3-bc26-c0004cf7433a admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:44.251403 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1143bd7-3bde-43c3-bc26-c0004cf7433a admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:44.251723 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1143bd7-3bde-43c3-bc26-c0004cf7433a admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:44.252341 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1143bd7-3bde-43c3-bc26-c0004cf7433a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:44.254290 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1143bd7-3bde-43c3-bc26-c0004cf7433a admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:44.287384 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1357/2729] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:53:44 2022] POST /identity/v3/projects => generated 373 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:53:44.291707 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-268a9a68-fe46-47df-aacb-d12787ecde3b admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:44.303379 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-268a9a68-fe46-47df-aacb-d12787ecde3b admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:44.370179 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b14d7b-ea07-4bed-8658-da425eadcfa3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:44.370938 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b14d7b-ea07-4bed-8658-da425eadcfa3 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:44.371256 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b14d7b-ea07-4bed-8658-da425eadcfa3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:44.371600 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b14d7b-ea07-4bed-8658-da425eadcfa3 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:44.372279 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b14d7b-ea07-4bed-8658-da425eadcfa3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:44.374353 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b14d7b-ea07-4bed-8658-da425eadcfa3 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:44.383594 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:44.398893 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1373/2730] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:53:44 2022] POST /identity/v3/users => generated 287 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:53:44.404133 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1143bd7-3bde-43c3-bc26-c0004cf7433a admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:44.415245 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1143bd7-3bde-43c3-bc26-c0004cf7433a admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:44.421231 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dc0e0d1-e2ce-4523-a625-2f5b29460d4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:44.422015 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc0e0d1-e2ce-4523-a625-2f5b29460d4b admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:44.422871 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc0e0d1-e2ce-4523-a625-2f5b29460d4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:44.423413 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc0e0d1-e2ce-4523-a625-2f5b29460d4b admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:44.424190 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dc0e0d1-e2ce-4523-a625-2f5b29460d4b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:44.426983 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dc0e0d1-e2ce-4523-a625-2f5b29460d4b admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:44.442384 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1358/2731] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:53:44 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:53:44.451906 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b14d7b-ea07-4bed-8658-da425eadcfa3 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:44.468323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b14d7b-ea07-4bed-8658-da425eadcfa3 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:44.475354 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d98c301-3a20-4663-80e9-8988069d7ffc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:44.476419 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d98c301-3a20-4663-80e9-8988069d7ffc admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:44.476914 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d98c301-3a20-4663-80e9-8988069d7ffc admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:44.477495 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d98c301-3a20-4663-80e9-8988069d7ffc admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:44.478605 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d98c301-3a20-4663-80e9-8988069d7ffc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:44.482551 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d98c301-3a20-4663-80e9-8988069d7ffc admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:44.500968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1374/2732] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:53:44 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:53:44.505980 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dc0e0d1-e2ce-4523-a625-2f5b29460d4b admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:44.517365 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dc0e0d1-e2ce-4523-a625-2f5b29460d4b admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:44.522302 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3edf9ba8-1ccb-4113-b3a6-a323a2af6e93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:44.523019 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3edf9ba8-1ccb-4113-b3a6-a323a2af6e93 admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:44.523301 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3edf9ba8-1ccb-4113-b3a6-a323a2af6e93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:44.523580 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3edf9ba8-1ccb-4113-b3a6-a323a2af6e93 admin admin] PATH_INFO: `/v3/projects/6737c612807a468797742417d59da976/users/a3c6fa6f402143b598108c6cb2aedf78/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:44.542097 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3edf9ba8-1ccb-4113-b3a6-a323a2af6e93 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6737c612807a468797742417d59da976, user_id=a3c6fa6f402143b598108c6cb2aedf78, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:44.545957 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3edf9ba8-1ccb-4113-b3a6-a323a2af6e93 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:44.576961 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1359/2733] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:53:44 2022] PUT /identity/v3/projects/6737c612807a468797742417d59da976/users/a3c6fa6f402143b598108c6cb2aedf78/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:44.586038 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f879d6-4178-4d1b-ab9a-6845ba0467df None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:44.586038 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f879d6-4178-4d1b-ab9a-6845ba0467df None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:44.586418 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f879d6-4178-4d1b-ab9a-6845ba0467df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:44.613145 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-89f879d6-4178-4d1b-ab9a-6845ba0467df None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:44.614428 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-89f879d6-4178-4d1b-ab9a-6845ba0467df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:44.694361 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1375/2734] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:53:44.705246 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3edf9ba8-1ccb-4113-b3a6-a323a2af6e93 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:44.707112 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-3edf9ba8-1ccb-4113-b3a6-a323a2af6e93 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:44.770675 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3edf9ba8-1ccb-4113-b3a6-a323a2af6e93 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:44.774431 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed676d3f-00fc-4095-9c96-39071222cdff service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:44.775074 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed676d3f-00fc-4095-9c96-39071222cdff service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:44.775352 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed676d3f-00fc-4095-9c96-39071222cdff service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:44.775626 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed676d3f-00fc-4095-9c96-39071222cdff service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:44.786168 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed676d3f-00fc-4095-9c96-39071222cdff service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:44.787819 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed676d3f-00fc-4095-9c96-39071222cdff service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:44.792098 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:44.799698 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1360/2735] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:53:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:53:45.789347 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bef2dd-c5d4-4fc7-b96e-15cfa9f25bb0 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:45.790198 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bef2dd-c5d4-4fc7-b96e-15cfa9f25bb0 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:45.790349 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bef2dd-c5d4-4fc7-b96e-15cfa9f25bb0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:45.830815 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-69bef2dd-c5d4-4fc7-b96e-15cfa9f25bb0 None None] MFA Rules not processed for user `a3c6fa6f402143b598108c6cb2aedf78`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:45.832238 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-69bef2dd-c5d4-4fc7-b96e-15cfa9f25bb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:45.918540 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1376/2736] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:45 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:53:45.936590 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed676d3f-00fc-4095-9c96-39071222cdff service neutron] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:45.938112 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-ed676d3f-00fc-4095-9c96-39071222cdff service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:46.001721 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed676d3f-00fc-4095-9c96-39071222cdff service neutron] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:46.005609 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692474ce-31d4-4416-9ff5-6a92424174ae service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:46.006247 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692474ce-31d4-4416-9ff5-6a92424174ae service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:46.006579 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692474ce-31d4-4416-9ff5-6a92424174ae service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:46.006963 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692474ce-31d4-4416-9ff5-6a92424174ae service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:46.017053 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-692474ce-31d4-4416-9ff5-6a92424174ae service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:46.018565 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-692474ce-31d4-4416-9ff5-6a92424174ae service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:46.030636 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1361/2737] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:53:45 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:53:49.636542 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b06961-9e00-40be-a81d-426f5ec353ea None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:49.637366 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b06961-9e00-40be-a81d-426f5ec353ea None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:49.638438 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b06961-9e00-40be-a81d-426f5ec353ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:49.662661 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-f3b06961-9e00-40be-a81d-426f5ec353ea None None] MFA Rules not processed for user `a3c6fa6f402143b598108c6cb2aedf78`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:49.663910 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-f3b06961-9e00-40be-a81d-426f5ec353ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:49.672633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1377/2738] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:49 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:53:52.883754 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692474ce-31d4-4416-9ff5-6a92424174ae service cinder] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:52.889157 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-692474ce-31d4-4416-9ff5-6a92424174ae service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:52.921776 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692474ce-31d4-4416-9ff5-6a92424174ae service cinder] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:52.931303 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06e00ba-f321-4c69-884d-371f82d460c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:52.933083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06e00ba-f321-4c69-884d-371f82d460c1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:52.933856 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06e00ba-f321-4c69-884d-371f82d460c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:52.934346 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06e00ba-f321-4c69-884d-371f82d460c1 admin admin] PATH_INFO: `/v3/users/f4c463cced614be791c68b47311bcc23` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:52.938209 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d06e00ba-f321-4c69-884d-371f82d460c1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f4c463cced614be791c68b47311bcc23)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:52.942619 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d06e00ba-f321-4c69-884d-371f82d460c1 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:53.046943 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-d06e00ba-f321-4c69-884d-371f82d460c1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f4c463cced614be791c68b47311bcc23'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:53.062261 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-d06e00ba-f321-4c69-884d-371f82d460c1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f4c463cced614be791c68b47311bcc23'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:53.063322 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-d06e00ba-f321-4c69-884d-371f82d460c1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f4c463cced614be791c68b47311bcc23'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:53.065382 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-d06e00ba-f321-4c69-884d-371f82d460c1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f4c463cced614be791c68b47311bcc23'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:53.076939 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-d06e00ba-f321-4c69-884d-371f82d460c1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f4c463cced614be791c68b47311bcc23'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:53.112080 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1362/2739] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:53:52 2022] DELETE /identity/v3/users/f4c463cced614be791c68b47311bcc23 => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:53.119465 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abeafeb5-0d54-4f72-a595-dc1f15bff55b None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:53.120075 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abeafeb5-0d54-4f72-a595-dc1f15bff55b None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:53.120673 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abeafeb5-0d54-4f72-a595-dc1f15bff55b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:53.155559 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-abeafeb5-0d54-4f72-a595-dc1f15bff55b None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:53.158040 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-abeafeb5-0d54-4f72-a595-dc1f15bff55b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:53.240366 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1378/2740] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:53:53.259514 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06e00ba-f321-4c69-884d-371f82d460c1 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:53.261197 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-d06e00ba-f321-4c69-884d-371f82d460c1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:53.341407 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06e00ba-f321-4c69-884d-371f82d460c1 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:53.345507 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c86b71f-70ac-4840-bbae-b88322500ae3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:53.346157 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c86b71f-70ac-4840-bbae-b88322500ae3 service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:53.346475 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c86b71f-70ac-4840-bbae-b88322500ae3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:53.346754 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c86b71f-70ac-4840-bbae-b88322500ae3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:53.358808 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c86b71f-70ac-4840-bbae-b88322500ae3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:53.360777 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c86b71f-70ac-4840-bbae-b88322500ae3 service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:53.376982 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1363/2741] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:53:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:53:53.969101 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abeafeb5-0d54-4f72-a595-dc1f15bff55b None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:53.970908 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-abeafeb5-0d54-4f72-a595-dc1f15bff55b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:53.980676 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:53.989692 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abeafeb5-0d54-4f72-a595-dc1f15bff55b None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:54.000527 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a05cc0-9e0d-44eb-9948-72e8fbc135c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:54.004232 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a05cc0-9e0d-44eb-9948-72e8fbc135c8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:54.005066 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a05cc0-9e0d-44eb-9948-72e8fbc135c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:54.006027 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a05cc0-9e0d-44eb-9948-72e8fbc135c8 admin admin] PATH_INFO: `/v3/projects/c724ddf152e44905b313c1976a4f1574` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:54.008578 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a05cc0-9e0d-44eb-9948-72e8fbc135c8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c724ddf152e44905b313c1976a4f1574)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:54.025430 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a05cc0-9e0d-44eb-9948-72e8fbc135c8 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:54.150014 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e5a05cc0-9e0d-44eb-9948-72e8fbc135c8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c724ddf152e44905b313c1976a4f1574', 'request_id': 'req-e5a05cc0-9e0d-44eb-9948-72e8fbc135c8'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:54.170564 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e5a05cc0-9e0d-44eb-9948-72e8fbc135c8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c724ddf152e44905b313c1976a4f1574', 'request_id': 'req-e5a05cc0-9e0d-44eb-9948-72e8fbc135c8'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:54.189623 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e5a05cc0-9e0d-44eb-9948-72e8fbc135c8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c724ddf152e44905b313c1976a4f1574', 'request_id': 'req-e5a05cc0-9e0d-44eb-9948-72e8fbc135c8'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:54.207000 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e5a05cc0-9e0d-44eb-9948-72e8fbc135c8 admin admin] The token cache is being invalidate because project c724ddf152e44905b313c1976a4f1574 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:53:54.207725 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e5a05cc0-9e0d-44eb-9948-72e8fbc135c8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:54.208830 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e5a05cc0-9e0d-44eb-9948-72e8fbc135c8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:54.212829 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1379/2742] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:53:53 2022] DELETE /identity/v3/projects/c724ddf152e44905b313c1976a4f1574 => generated 0 bytes in 244 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:53:54.220062 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c86b71f-70ac-4840-bbae-b88322500ae3 service neutron] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:54.242768 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c86b71f-70ac-4840-bbae-b88322500ae3 service neutron] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:54.268763 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:54.345208 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261aba94-127c-4c23-a8aa-555ec108bd2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:54.345999 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261aba94-127c-4c23-a8aa-555ec108bd2d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:54.346439 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261aba94-127c-4c23-a8aa-555ec108bd2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:54.346743 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261aba94-127c-4c23-a8aa-555ec108bd2d admin admin] PATH_INFO: `/v3/users/cbc121319cce43ff897a367465693897` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:54.348202 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-261aba94-127c-4c23-a8aa-555ec108bd2d admin admin] RBAC: Authorizing `identity:delete_user(user_id=cbc121319cce43ff897a367465693897)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:54.350923 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-261aba94-127c-4c23-a8aa-555ec108bd2d admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:54.421455 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-261aba94-127c-4c23-a8aa-555ec108bd2d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cbc121319cce43ff897a367465693897'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:54.436958 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-261aba94-127c-4c23-a8aa-555ec108bd2d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cbc121319cce43ff897a367465693897'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:54.438051 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-261aba94-127c-4c23-a8aa-555ec108bd2d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cbc121319cce43ff897a367465693897'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:54.439099 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-261aba94-127c-4c23-a8aa-555ec108bd2d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cbc121319cce43ff897a367465693897'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:54.451933 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-261aba94-127c-4c23-a8aa-555ec108bd2d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cbc121319cce43ff897a367465693897'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:54.474377 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1364/2743] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:53:54 2022] DELETE /identity/v3/users/cbc121319cce43ff897a367465693897 => generated 0 bytes in 256 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:54.883902 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a05cc0-9e0d-44eb-9948-72e8fbc135c8 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:54.886279 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e5a05cc0-9e0d-44eb-9948-72e8fbc135c8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:54.910575 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a05cc0-9e0d-44eb-9948-72e8fbc135c8 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:55.003483 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e007116e-26ab-4e53-8745-827e5c801c00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:55.004142 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e007116e-26ab-4e53-8745-827e5c801c00 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:55.004430 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e007116e-26ab-4e53-8745-827e5c801c00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:55.004704 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e007116e-26ab-4e53-8745-827e5c801c00 admin admin] PATH_INFO: `/v3/projects/bb30290b9a8a467a81ef70f6949da18b` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:55.005953 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e007116e-26ab-4e53-8745-827e5c801c00 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bb30290b9a8a467a81ef70f6949da18b)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:55.007922 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e007116e-26ab-4e53-8745-827e5c801c00 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:55.073341 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e007116e-26ab-4e53-8745-827e5c801c00 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bb30290b9a8a467a81ef70f6949da18b', 'request_id': 'req-e007116e-26ab-4e53-8745-827e5c801c00'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:55.081611 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e007116e-26ab-4e53-8745-827e5c801c00 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bb30290b9a8a467a81ef70f6949da18b', 'request_id': 'req-e007116e-26ab-4e53-8745-827e5c801c00'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:55.091505 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e007116e-26ab-4e53-8745-827e5c801c00 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bb30290b9a8a467a81ef70f6949da18b', 'request_id': 'req-e007116e-26ab-4e53-8745-827e5c801c00'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:55.104779 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e007116e-26ab-4e53-8745-827e5c801c00 admin admin] The token cache is being invalidate because project bb30290b9a8a467a81ef70f6949da18b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:53:55.105226 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e007116e-26ab-4e53-8745-827e5c801c00 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:55.106066 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e007116e-26ab-4e53-8745-827e5c801c00 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:55.108352 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1380/2744] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:53:54 2022] DELETE /identity/v3/projects/bb30290b9a8a467a81ef70f6949da18b => generated 0 bytes in 226 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:55.113783 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2309781-ccc6-44cd-b481-b3d46e39f700 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:55.114092 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2309781-ccc6-44cd-b481-b3d46e39f700 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:55.114376 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2309781-ccc6-44cd-b481-b3d46e39f700 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:55.135964 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-c2309781-ccc6-44cd-b481-b3d46e39f700 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:55.137228 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-c2309781-ccc6-44cd-b481-b3d46e39f700 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:55.205288 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1365/2745] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:53:55.213889 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccfa641-b4a1-44e7-8b3e-1ab096ddb5e6 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:55.214592 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccfa641-b4a1-44e7-8b3e-1ab096ddb5e6 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:55.214592 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccfa641-b4a1-44e7-8b3e-1ab096ddb5e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:55.236805 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-6ccfa641-b4a1-44e7-8b3e-1ab096ddb5e6 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:55.238057 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-6ccfa641-b4a1-44e7-8b3e-1ab096ddb5e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:55.248044 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1381/2746] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:53:55.253682 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2309781-ccc6-44cd-b481-b3d46e39f700 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:55.272159 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2309781-ccc6-44cd-b481-b3d46e39f700 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:55.280626 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f68ca3a2-5364-4fcb-848c-504ac8a47d44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:55.281643 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68ca3a2-5364-4fcb-848c-504ac8a47d44 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:55.282117 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68ca3a2-5364-4fcb-848c-504ac8a47d44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:55.282591 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68ca3a2-5364-4fcb-848c-504ac8a47d44 admin admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:55.283654 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f68ca3a2-5364-4fcb-848c-504ac8a47d44 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:53:55.284188 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f68ca3a2-5364-4fcb-848c-504ac8a47d44 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:55.287108 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f68ca3a2-5364-4fcb-848c-504ac8a47d44 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:55.308655 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1366/2747] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:53:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:53:55.313978 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ccfa641-b4a1-44e7-8b3e-1ab096ddb5e6 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:55.324537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ccfa641-b4a1-44e7-8b3e-1ab096ddb5e6 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:55.330066 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98160304-87ec-45ad-80ed-ae8dcc9d5fe0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:55.330741 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98160304-87ec-45ad-80ed-ae8dcc9d5fe0 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:55.331021 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98160304-87ec-45ad-80ed-ae8dcc9d5fe0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:55.331294 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98160304-87ec-45ad-80ed-ae8dcc9d5fe0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:55.331944 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98160304-87ec-45ad-80ed-ae8dcc9d5fe0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:55.333880 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98160304-87ec-45ad-80ed-ae8dcc9d5fe0 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:55.340392 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:55.367430 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1382/2748] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:53:55 2022] POST /identity/v3/projects => generated 381 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:53:55.372837 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f68ca3a2-5364-4fcb-848c-504ac8a47d44 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:55.385162 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f68ca3a2-5364-4fcb-848c-504ac8a47d44 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:55.410409 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:55.443131 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98160304-87ec-45ad-80ed-ae8dcc9d5fe0 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:55.445181 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-98160304-87ec-45ad-80ed-ae8dcc9d5fe0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:55.467909 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9221f527-b13e-441c-bbc1-60b7d3d83b01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:55.468431 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9221f527-b13e-441c-bbc1-60b7d3d83b01 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:55.468712 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9221f527-b13e-441c-bbc1-60b7d3d83b01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:55.468991 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9221f527-b13e-441c-bbc1-60b7d3d83b01 admin admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:55.469647 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9221f527-b13e-441c-bbc1-60b7d3d83b01 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:55.471709 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9221f527-b13e-441c-bbc1-60b7d3d83b01 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:55.498881 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1367/2749] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:53:55 2022] POST /identity/v3/users => generated 291 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:53:55.504075 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9221f527-b13e-441c-bbc1-60b7d3d83b01 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:55.517460 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9221f527-b13e-441c-bbc1-60b7d3d83b01 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:55.523359 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-098f8508-e4ff-43cd-ae90-6f7ece9ddfe1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:55.524170 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098f8508-e4ff-43cd-ae90-6f7ece9ddfe1 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:55.524629 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098f8508-e4ff-43cd-ae90-6f7ece9ddfe1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:55.525118 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098f8508-e4ff-43cd-ae90-6f7ece9ddfe1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:55.526025 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-098f8508-e4ff-43cd-ae90-6f7ece9ddfe1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:55.527930 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-098f8508-e4ff-43cd-ae90-6f7ece9ddfe1 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:55.531427 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98160304-87ec-45ad-80ed-ae8dcc9d5fe0 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:55.537796 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6c36a97-424a-4801-872d-a0d44aec923f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:55.538654 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c36a97-424a-4801-872d-a0d44aec923f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:55.538998 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c36a97-424a-4801-872d-a0d44aec923f admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:55.539370 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6c36a97-424a-4801-872d-a0d44aec923f admin admin] PATH_INFO: `/v3/users/1010cf91e3d14a74925444d66ae47663` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:55.540758 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c36a97-424a-4801-872d-a0d44aec923f admin admin] RBAC: Authorizing `identity:delete_user(user_id=1010cf91e3d14a74925444d66ae47663)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:55.540974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1368/2750] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:53:55 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:53:55.543149 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6c36a97-424a-4801-872d-a0d44aec923f admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:55.545261 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-098f8508-e4ff-43cd-ae90-6f7ece9ddfe1 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:55.555957 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-098f8508-e4ff-43cd-ae90-6f7ece9ddfe1 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:55.561178 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23266e26-0e85-45b4-a419-0745b9aa0df8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:55.562175 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23266e26-0e85-45b4-a419-0745b9aa0df8 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:55.562175 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23266e26-0e85-45b4-a419-0745b9aa0df8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:55.562527 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23266e26-0e85-45b4-a419-0745b9aa0df8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:55.563113 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23266e26-0e85-45b4-a419-0745b9aa0df8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:55.564768 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23266e26-0e85-45b4-a419-0745b9aa0df8 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:55.577105 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1369/2751] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:53:55 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:53:55.581618 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23266e26-0e85-45b4-a419-0745b9aa0df8 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:55.595499 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23266e26-0e85-45b4-a419-0745b9aa0df8 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:55.603502 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0bfa76-ba74-4471-a704-c0f33b66dd2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:55.605918 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0bfa76-ba74-4471-a704-c0f33b66dd2e admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:55.606559 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0bfa76-ba74-4471-a704-c0f33b66dd2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:55.606922 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0bfa76-ba74-4471-a704-c0f33b66dd2e admin admin] PATH_INFO: `/v3/projects/f0223b014e5a4a6284f2babb5bbf3ace/users/2a25ebb07ec64e5590a4a3ce3150b779/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:55.628415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a6c36a97-424a-4801-872d-a0d44aec923f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1010cf91e3d14a74925444d66ae47663'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:55.633216 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0bfa76-ba74-4471-a704-c0f33b66dd2e admin admin] RBAC: Authorizing `identity:create_grant(project_id=f0223b014e5a4a6284f2babb5bbf3ace, user_id=2a25ebb07ec64e5590a4a3ce3150b779, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:55.639433 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0bfa76-ba74-4471-a704-c0f33b66dd2e admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:55.641676 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a6c36a97-424a-4801-872d-a0d44aec923f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1010cf91e3d14a74925444d66ae47663'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:55.642806 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a6c36a97-424a-4801-872d-a0d44aec923f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1010cf91e3d14a74925444d66ae47663'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:55.643993 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a6c36a97-424a-4801-872d-a0d44aec923f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1010cf91e3d14a74925444d66ae47663'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:55.657704 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a6c36a97-424a-4801-872d-a0d44aec923f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1010cf91e3d14a74925444d66ae47663'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:55.666991 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1370/2752] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:53:55 2022] PUT /identity/v3/projects/f0223b014e5a4a6284f2babb5bbf3ace/users/2a25ebb07ec64e5590a4a3ce3150b779/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 86 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:53:55.687550 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1383/2753] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:53:55 2022] DELETE /identity/v3/users/1010cf91e3d14a74925444d66ae47663 => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:53:55.688660 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0e5104-1917-4600-bb59-9f4370c1650d None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:55.689124 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0e5104-1917-4600-bb59-9f4370c1650d None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:55.689600 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0e5104-1917-4600-bb59-9f4370c1650d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:55.693252 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046f8d17-6a8f-4821-a852-bf244fcfb77f None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:55.693713 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046f8d17-6a8f-4821-a852-bf244fcfb77f None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:55.694142 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046f8d17-6a8f-4821-a852-bf244fcfb77f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:55.717790 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-046f8d17-6a8f-4821-a852-bf244fcfb77f None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:55.719170 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-046f8d17-6a8f-4821-a852-bf244fcfb77f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:55.738928 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-9a0e5104-1917-4600-bb59-9f4370c1650d None None] MFA Rules not processed for user `2a25ebb07ec64e5590a4a3ce3150b779`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:55.740203 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-9a0e5104-1917-4600-bb59-9f4370c1650d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:55.806629 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1384/2754] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:53:55.821001 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046f8d17-6a8f-4821-a852-bf244fcfb77f None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:55.822474 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-046f8d17-6a8f-4821-a852-bf244fcfb77f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:55.837122 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:55.856315 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1371/2755] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:55 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:53:55.862523 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a0e5104-1917-4600-bb59-9f4370c1650d None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:55.864591 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-9a0e5104-1917-4600-bb59-9f4370c1650d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:55.884064 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a0e5104-1917-4600-bb59-9f4370c1650d None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:55.890177 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe06789a-a3ea-4b16-a53b-6962fa4e4c73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:55.891011 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe06789a-a3ea-4b16-a53b-6962fa4e4c73 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:55.891360 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe06789a-a3ea-4b16-a53b-6962fa4e4c73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:55.891703 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe06789a-a3ea-4b16-a53b-6962fa4e4c73 admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:55.892428 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe06789a-a3ea-4b16-a53b-6962fa4e4c73 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:55.894575 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe06789a-a3ea-4b16-a53b-6962fa4e4c73 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:55.930567 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1372/2756] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:53:55 2022] POST /identity/v3/projects => generated 381 bytes in 69 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:53:55.935380 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe06789a-a3ea-4b16-a53b-6962fa4e4c73 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:55.953020 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046f8d17-6a8f-4821-a852-bf244fcfb77f None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:55.991283 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:56.052685 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe06789a-a3ea-4b16-a53b-6962fa4e4c73 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:56.061686 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-259dab58-55c8-4c0e-a0b3-8218e255079c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:56.062855 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259dab58-55c8-4c0e-a0b3-8218e255079c admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:56.063265 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259dab58-55c8-4c0e-a0b3-8218e255079c admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:56.063758 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259dab58-55c8-4c0e-a0b3-8218e255079c admin admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:56.064939 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-259dab58-55c8-4c0e-a0b3-8218e255079c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:56.068809 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-259dab58-55c8-4c0e-a0b3-8218e255079c admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:56.072049 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d0bc742-08c9-4d83-a618-fdf325b61f1f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:56.073130 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0bc742-08c9-4d83-a618-fdf325b61f1f service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:56.073644 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0bc742-08c9-4d83-a618-fdf325b61f1f service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:56.074133 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0bc742-08c9-4d83-a618-fdf325b61f1f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:56.089638 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d0bc742-08c9-4d83-a618-fdf325b61f1f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:56.092881 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d0bc742-08c9-4d83-a618-fdf325b61f1f service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:56.109878 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1373/2757] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:53:55 2022] POST /identity/v3/users => generated 297 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:53:56.111967 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-259dab58-55c8-4c0e-a0b3-8218e255079c admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:56.124980 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1385/2758] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:53:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:53:56.131120 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-259dab58-55c8-4c0e-a0b3-8218e255079c admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:56.139033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303d7480-6f77-4d0f-a0b7-05f3a1ec1d73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:56.140104 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303d7480-6f77-4d0f-a0b7-05f3a1ec1d73 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:56.140604 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303d7480-6f77-4d0f-a0b7-05f3a1ec1d73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:56.141109 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303d7480-6f77-4d0f-a0b7-05f3a1ec1d73 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:56.142348 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-303d7480-6f77-4d0f-a0b7-05f3a1ec1d73 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:56.145540 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-303d7480-6f77-4d0f-a0b7-05f3a1ec1d73 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:56.164952 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1374/2759] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:53:56 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:53:56.165991 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d0bc742-08c9-4d83-a618-fdf325b61f1f service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:56.179421 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d0bc742-08c9-4d83-a618-fdf325b61f1f service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:56.186578 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd92c1e-fab8-4a2f-bd47-5e181e82f1b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:56.187404 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd92c1e-fab8-4a2f-bd47-5e181e82f1b4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:56.187754 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd92c1e-fab8-4a2f-bd47-5e181e82f1b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:56.188106 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd92c1e-fab8-4a2f-bd47-5e181e82f1b4 admin admin] PATH_INFO: `/v3/projects/b288dcf97b964751b4cff1cc6460b022/users/129eeefcec1e4136aba36eb6425a3c3c/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:56.212311 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd92c1e-fab8-4a2f-bd47-5e181e82f1b4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b288dcf97b964751b4cff1cc6460b022, user_id=129eeefcec1e4136aba36eb6425a3c3c, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:56.216287 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd92c1e-fab8-4a2f-bd47-5e181e82f1b4 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:56.242793 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1386/2760] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:53:56 2022] PUT /identity/v3/projects/b288dcf97b964751b4cff1cc6460b022/users/129eeefcec1e4136aba36eb6425a3c3c/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:56.244767 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303d7480-6f77-4d0f-a0b7-05f3a1ec1d73 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:56.258074 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303d7480-6f77-4d0f-a0b7-05f3a1ec1d73 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:56.342615 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-833c2a31-742a-45c1-9c1a-5bdd5fc765d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:56.343278 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833c2a31-742a-45c1-9c1a-5bdd5fc765d6 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:56.343560 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833c2a31-742a-45c1-9c1a-5bdd5fc765d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:56.343859 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-833c2a31-742a-45c1-9c1a-5bdd5fc765d6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:56.344516 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-833c2a31-742a-45c1-9c1a-5bdd5fc765d6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:56.346672 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-833c2a31-742a-45c1-9c1a-5bdd5fc765d6 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:56.359987 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1375/2761] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:53:56 2022] GET /identity/v3/roles => generated 1420 bytes in 116 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:53:56.365165 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd92c1e-fab8-4a2f-bd47-5e181e82f1b4 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:56.383948 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd92c1e-fab8-4a2f-bd47-5e181e82f1b4 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:56.397520 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7fb9e61-ba67-469a-bec4-59e8dc8fd763 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:56.398701 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7fb9e61-ba67-469a-bec4-59e8dc8fd763 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:56.399186 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7fb9e61-ba67-469a-bec4-59e8dc8fd763 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:56.399716 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7fb9e61-ba67-469a-bec4-59e8dc8fd763 admin admin] PATH_INFO: `/v3/projects/b288dcf97b964751b4cff1cc6460b022/users/129eeefcec1e4136aba36eb6425a3c3c/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:56.404215 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7fb9e61-ba67-469a-bec4-59e8dc8fd763 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b288dcf97b964751b4cff1cc6460b022, user_id=129eeefcec1e4136aba36eb6425a3c3c, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:56.409799 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7fb9e61-ba67-469a-bec4-59e8dc8fd763 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:56.431207 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1387/2762] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:53:56 2022] PUT /identity/v3/projects/b288dcf97b964751b4cff1cc6460b022/users/129eeefcec1e4136aba36eb6425a3c3c/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:56.448743 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb0bf9b8-fdc0-4709-9ac7-e32e5e6c3ad8 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:56.449356 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb0bf9b8-fdc0-4709-9ac7-e32e5e6c3ad8 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:56.449958 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb0bf9b8-fdc0-4709-9ac7-e32e5e6c3ad8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:56.503923 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-bb0bf9b8-fdc0-4709-9ac7-e32e5e6c3ad8 None None] MFA Rules not processed for user `129eeefcec1e4136aba36eb6425a3c3c`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:56.506476 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-bb0bf9b8-fdc0-4709-9ac7-e32e5e6c3ad8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:56.549962 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7fb9e61-ba67-469a-bec4-59e8dc8fd763 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:56.552641 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-b7fb9e61-ba67-469a-bec4-59e8dc8fd763 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:56.649873 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1376/2763] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:56 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:53:56.661994 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb0bf9b8-fdc0-4709-9ac7-e32e5e6c3ad8 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:56.663460 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-bb0bf9b8-fdc0-4709-9ac7-e32e5e6c3ad8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:56.668241 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7fb9e61-ba67-469a-bec4-59e8dc8fd763 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:56.674022 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4dd1190-ee88-43a3-aa13-7f3b353ad023 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:56.674722 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4dd1190-ee88-43a3-aa13-7f3b353ad023 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:56.674982 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4dd1190-ee88-43a3-aa13-7f3b353ad023 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:56.675266 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4dd1190-ee88-43a3-aa13-7f3b353ad023 admin admin] PATH_INFO: `/v3/projects/8fd998b657cf4b9c92fdbd73f0d7acce` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:56.676672 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4dd1190-ee88-43a3-aa13-7f3b353ad023 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8fd998b657cf4b9c92fdbd73f0d7acce)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:56.678610 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4dd1190-ee88-43a3-aa13-7f3b353ad023 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:56.708411 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:56.732850 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=96,keystone:INSERT=2,keystone:DELETE=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:56.743706 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb0bf9b8-fdc0-4709-9ac7-e32e5e6c3ad8 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:56.758795 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c4dd1190-ee88-43a3-aa13-7f3b353ad023 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8fd998b657cf4b9c92fdbd73f0d7acce', 'request_id': 'req-c4dd1190-ee88-43a3-aa13-7f3b353ad023'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:56.770913 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c4dd1190-ee88-43a3-aa13-7f3b353ad023 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8fd998b657cf4b9c92fdbd73f0d7acce', 'request_id': 'req-c4dd1190-ee88-43a3-aa13-7f3b353ad023'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:56.784158 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c4dd1190-ee88-43a3-aa13-7f3b353ad023 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8fd998b657cf4b9c92fdbd73f0d7acce', 'request_id': 'req-c4dd1190-ee88-43a3-aa13-7f3b353ad023'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:56.798227 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c4dd1190-ee88-43a3-aa13-7f3b353ad023 admin admin] The token cache is being invalidate because project 8fd998b657cf4b9c92fdbd73f0d7acce was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:53:56.798736 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c4dd1190-ee88-43a3-aa13-7f3b353ad023 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:56.799682 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-c4dd1190-ee88-43a3-aa13-7f3b353ad023 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:56.802037 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1388/2764] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:53:56 2022] DELETE /identity/v3/projects/8fd998b657cf4b9c92fdbd73f0d7acce => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:53:56.808300 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107a32a8-e8ff-41c5-8d3c-9ee855ccce10 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:56.809032 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107a32a8-e8ff-41c5-8d3c-9ee855ccce10 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:56.809606 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107a32a8-e8ff-41c5-8d3c-9ee855ccce10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:56.813884 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2a0b9da-d24a-4a5b-86fb-6adb746b02f9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:56.814565 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a0b9da-d24a-4a5b-86fb-6adb746b02f9 service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:56.814848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a0b9da-d24a-4a5b-86fb-6adb746b02f9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:56.815119 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a0b9da-d24a-4a5b-86fb-6adb746b02f9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:56.816928 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-a2a0b9da-d24a-4a5b-86fb-6adb746b02f9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:56.840332 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-107a32a8-e8ff-41c5-8d3c-9ee855ccce10 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:56.843099 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-107a32a8-e8ff-41c5-8d3c-9ee855ccce10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:56.900068 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2a0b9da-d24a-4a5b-86fb-6adb746b02f9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:56.901962 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2a0b9da-d24a-4a5b-86fb-6adb746b02f9 service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:56.917698 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1377/2765] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:53:56 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 256 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:53:56.918288 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1389/2766] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:53:56.927742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7879cbd2-5c47-49b2-b783-48f6f8b75abe None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:56.928231 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7879cbd2-5c47-49b2-b783-48f6f8b75abe None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:56.928689 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7879cbd2-5c47-49b2-b783-48f6f8b75abe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:56.961802 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-7879cbd2-5c47-49b2-b783-48f6f8b75abe None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:56.963963 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-7879cbd2-5c47-49b2-b783-48f6f8b75abe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:56.983461 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1378/2767] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:53:56.985196 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-107a32a8-e8ff-41c5-8d3c-9ee855ccce10 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:56.999408 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-107a32a8-e8ff-41c5-8d3c-9ee855ccce10 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:57.007581 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3e63c0-825c-428f-af54-bb20f2a5eddb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:57.008591 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3e63c0-825c-428f-af54-bb20f2a5eddb admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:57.009124 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3e63c0-825c-428f-af54-bb20f2a5eddb admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:57.009492 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c3e63c0-825c-428f-af54-bb20f2a5eddb admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:57.010352 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c3e63c0-825c-428f-af54-bb20f2a5eddb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:53:57.010873 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c3e63c0-825c-428f-af54-bb20f2a5eddb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:57.014835 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c3e63c0-825c-428f-af54-bb20f2a5eddb admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:57.039972 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1390/2768] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:53:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:53:57.041976 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7879cbd2-5c47-49b2-b783-48f6f8b75abe None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:57.062603 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7879cbd2-5c47-49b2-b783-48f6f8b75abe None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:57.064402 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149fb5f1-1407-4b24-b090-5fb87e7fb925 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:57.065387 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149fb5f1-1407-4b24-b090-5fb87e7fb925 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:57.065778 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149fb5f1-1407-4b24-b090-5fb87e7fb925 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:57.066243 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149fb5f1-1407-4b24-b090-5fb87e7fb925 admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:57.067393 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-149fb5f1-1407-4b24-b090-5fb87e7fb925 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:57.070793 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-149fb5f1-1407-4b24-b090-5fb87e7fb925 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:57.130598 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1379/2769] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:53:57 2022] POST /identity/v3/projects => generated 375 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:53:57.136426 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3e63c0-825c-428f-af54-bb20f2a5eddb admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:57.158785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c3e63c0-825c-428f-af54-bb20f2a5eddb admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:57.254164 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98fabd3e-c3f5-42b3-b55c-b90d34b02d55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:57.255346 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98fabd3e-c3f5-42b3-b55c-b90d34b02d55 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:57.255346 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98fabd3e-c3f5-42b3-b55c-b90d34b02d55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:57.255696 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98fabd3e-c3f5-42b3-b55c-b90d34b02d55 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:57.256176 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98fabd3e-c3f5-42b3-b55c-b90d34b02d55 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:57.258139 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98fabd3e-c3f5-42b3-b55c-b90d34b02d55 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:57.283339 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1391/2770] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:53:57 2022] POST /identity/v3/users => generated 288 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:53:57.289672 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149fb5f1-1407-4b24-b090-5fb87e7fb925 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:57.304909 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149fb5f1-1407-4b24-b090-5fb87e7fb925 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:57.313610 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c1450dc-353f-42c9-9fef-c1dc8be38ef1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:57.314901 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c1450dc-353f-42c9-9fef-c1dc8be38ef1 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:57.315374 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c1450dc-353f-42c9-9fef-c1dc8be38ef1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:57.315846 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c1450dc-353f-42c9-9fef-c1dc8be38ef1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:57.317064 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c1450dc-353f-42c9-9fef-c1dc8be38ef1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:57.320063 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c1450dc-353f-42c9-9fef-c1dc8be38ef1 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:57.343394 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1380/2771] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:53:57 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:53:57.346249 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98fabd3e-c3f5-42b3-b55c-b90d34b02d55 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:57.363990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98fabd3e-c3f5-42b3-b55c-b90d34b02d55 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:57.372877 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b2f9343-7f4e-4a8e-9e0a-9e7784ea6697 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:57.373931 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2f9343-7f4e-4a8e-9e0a-9e7784ea6697 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:57.374464 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2f9343-7f4e-4a8e-9e0a-9e7784ea6697 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:57.375015 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2f9343-7f4e-4a8e-9e0a-9e7784ea6697 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:57.376177 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b2f9343-7f4e-4a8e-9e0a-9e7784ea6697 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:57.379344 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b2f9343-7f4e-4a8e-9e0a-9e7784ea6697 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:57.402594 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1392/2772] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:53:57 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:53:57.404880 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c1450dc-353f-42c9-9fef-c1dc8be38ef1 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:57.422200 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c1450dc-353f-42c9-9fef-c1dc8be38ef1 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:57.430606 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39eb2666-7b40-4990-889b-c63806b35712 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:57.431754 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39eb2666-7b40-4990-889b-c63806b35712 admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:57.432238 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39eb2666-7b40-4990-889b-c63806b35712 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:57.432715 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39eb2666-7b40-4990-889b-c63806b35712 admin admin] PATH_INFO: `/v3/projects/1d832ea68ebb4c60b962d0cd76f787b8/users/4d088636881e45559699462268bd0411/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:57.441951 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b2f9343-7f4e-4a8e-9e0a-9e7784ea6697 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:57.460685 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39eb2666-7b40-4990-889b-c63806b35712 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1d832ea68ebb4c60b962d0cd76f787b8, user_id=4d088636881e45559699462268bd0411, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:57.463987 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39eb2666-7b40-4990-889b-c63806b35712 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:57.475534 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:57.483946 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1381/2773] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:53:57 2022] PUT /identity/v3/projects/1d832ea68ebb4c60b962d0cd76f787b8/users/4d088636881e45559699462268bd0411/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 80 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:57.495576 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edebd05e-9b53-4427-a3e5-08c6cbfea9f5 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:57.496037 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edebd05e-9b53-4427-a3e5-08c6cbfea9f5 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:57.496481 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edebd05e-9b53-4427-a3e5-08c6cbfea9f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:57.541309 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b2f9343-7f4e-4a8e-9e0a-9e7784ea6697 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:57.543160 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-edebd05e-9b53-4427-a3e5-08c6cbfea9f5 None None] MFA Rules not processed for user `4d088636881e45559699462268bd0411`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:57.545684 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-edebd05e-9b53-4427-a3e5-08c6cbfea9f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:57.594742 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:57.632895 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b902a39b-0be2-4cf2-b69b-b747fd84ca5a tempest-VolumeQuotaClassesTest-1642285887 tempest-VolumeQuotaClassesTest-1642285887-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '129eeefcec1e4136aba36eb6425a3c3c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b288dcf97b964751b4cff1cc6460b022', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:57.633897 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b902a39b-0be2-4cf2-b69b-b747fd84ca5a tempest-VolumeQuotaClassesTest-1642285887 tempest-VolumeQuotaClassesTest-1642285887-project-admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:57.634478 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b902a39b-0be2-4cf2-b69b-b747fd84ca5a tempest-VolumeQuotaClassesTest-1642285887 tempest-VolumeQuotaClassesTest-1642285887-project-admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:57.635056 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b902a39b-0be2-4cf2-b69b-b747fd84ca5a tempest-VolumeQuotaClassesTest-1642285887 tempest-VolumeQuotaClassesTest-1642285887-project-admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:57.636158 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b902a39b-0be2-4cf2-b69b-b747fd84ca5a tempest-VolumeQuotaClassesTest-1642285887 tempest-VolumeQuotaClassesTest-1642285887-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:53:57.636719 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b902a39b-0be2-4cf2-b69b-b747fd84ca5a tempest-VolumeQuotaClassesTest-1642285887 tempest-VolumeQuotaClassesTest-1642285887-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:57.639847 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b902a39b-0be2-4cf2-b69b-b747fd84ca5a tempest-VolumeQuotaClassesTest-1642285887 tempest-VolumeQuotaClassesTest-1642285887-project-admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:57.662636 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1393/2774] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:53:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:53:57.665961 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1382/2775] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:57 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:53:57.666677 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b902a39b-0be2-4cf2-b69b-b747fd84ca5a tempest-VolumeQuotaClassesTest-1642285887 tempest-VolumeQuotaClassesTest-1642285887-project-admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:57.679482 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edebd05e-9b53-4427-a3e5-08c6cbfea9f5 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:57.681130 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-edebd05e-9b53-4427-a3e5-08c6cbfea9f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:57.683265 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b902a39b-0be2-4cf2-b69b-b747fd84ca5a tempest-VolumeQuotaClassesTest-1642285887 tempest-VolumeQuotaClassesTest-1642285887-project-admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:57.692380 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d490e7b-1fd5-4128-bc8f-a5d626eb85ee tempest-VolumeQuotaClassesTest-1642285887 tempest-VolumeQuotaClassesTest-1642285887-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '129eeefcec1e4136aba36eb6425a3c3c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b288dcf97b964751b4cff1cc6460b022', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:57.693428 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d490e7b-1fd5-4128-bc8f-a5d626eb85ee tempest-VolumeQuotaClassesTest-1642285887 tempest-VolumeQuotaClassesTest-1642285887-project-admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:57.693949 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d490e7b-1fd5-4128-bc8f-a5d626eb85ee tempest-VolumeQuotaClassesTest-1642285887 tempest-VolumeQuotaClassesTest-1642285887-project-admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:57.694522 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d490e7b-1fd5-4128-bc8f-a5d626eb85ee tempest-VolumeQuotaClassesTest-1642285887 tempest-VolumeQuotaClassesTest-1642285887-project-admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:57.695734 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d490e7b-1fd5-4128-bc8f-a5d626eb85ee tempest-VolumeQuotaClassesTest-1642285887 tempest-VolumeQuotaClassesTest-1642285887-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:57.699598 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d490e7b-1fd5-4128-bc8f-a5d626eb85ee tempest-VolumeQuotaClassesTest-1642285887 tempest-VolumeQuotaClassesTest-1642285887-project-admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:57.747962 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1394/2776] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:53:57 2022] POST /identity/v3/projects => generated 355 bytes in 82 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:53:57.754340 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d490e7b-1fd5-4128-bc8f-a5d626eb85ee tempest-VolumeQuotaClassesTest-1642285887 tempest-VolumeQuotaClassesTest-1642285887-project-admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:57.760248 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edebd05e-9b53-4427-a3e5-08c6cbfea9f5 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:57.832566 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38228714-730d-4afb-87ff-1c6fa8c42241 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:57.833450 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38228714-730d-4afb-87ff-1c6fa8c42241 service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:57.833883 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38228714-730d-4afb-87ff-1c6fa8c42241 service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:57.834298 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38228714-730d-4afb-87ff-1c6fa8c42241 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:57.842794 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d490e7b-1fd5-4128-bc8f-a5d626eb85ee tempest-VolumeQuotaClassesTest-1642285887 tempest-VolumeQuotaClassesTest-1642285887-project-admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:57.848253 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-293abec2-beb2-43b4-87d0-a1618b73ccb8 tempest-VolumeQuotaClassesTest-1642285887 tempest-VolumeQuotaClassesTest-1642285887-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '129eeefcec1e4136aba36eb6425a3c3c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b288dcf97b964751b4cff1cc6460b022', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:57.849217 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293abec2-beb2-43b4-87d0-a1618b73ccb8 tempest-VolumeQuotaClassesTest-1642285887 tempest-VolumeQuotaClassesTest-1642285887-project-admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:57.849668 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293abec2-beb2-43b4-87d0-a1618b73ccb8 tempest-VolumeQuotaClassesTest-1642285887 tempest-VolumeQuotaClassesTest-1642285887-project-admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:57.850038 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293abec2-beb2-43b4-87d0-a1618b73ccb8 tempest-VolumeQuotaClassesTest-1642285887 tempest-VolumeQuotaClassesTest-1642285887-project-admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:57.851116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-293abec2-beb2-43b4-87d0-a1618b73ccb8 tempest-VolumeQuotaClassesTest-1642285887 tempest-VolumeQuotaClassesTest-1642285887-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:53:57.851642 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-293abec2-beb2-43b4-87d0-a1618b73ccb8 tempest-VolumeQuotaClassesTest-1642285887 tempest-VolumeQuotaClassesTest-1642285887-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:57.853780 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-293abec2-beb2-43b4-87d0-a1618b73ccb8 tempest-VolumeQuotaClassesTest-1642285887 tempest-VolumeQuotaClassesTest-1642285887-project-admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:57.868074 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1395/2777] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:53:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 115 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:53:57.912566 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-293abec2-beb2-43b4-87d0-a1618b73ccb8 tempest-VolumeQuotaClassesTest-1642285887 tempest-VolumeQuotaClassesTest-1642285887-project-admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:57.929382 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-293abec2-beb2-43b4-87d0-a1618b73ccb8 tempest-VolumeQuotaClassesTest-1642285887 tempest-VolumeQuotaClassesTest-1642285887-project-admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:57.930191 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38228714-730d-4afb-87ff-1c6fa8c42241 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:57.931908 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38228714-730d-4afb-87ff-1c6fa8c42241 service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:57.938048 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25026cb4-bc01-4d0a-837d-fa4c8d45cfbb tempest-VolumeQuotaClassesTest-1642285887 tempest-VolumeQuotaClassesTest-1642285887-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '129eeefcec1e4136aba36eb6425a3c3c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'b288dcf97b964751b4cff1cc6460b022', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:57.939186 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25026cb4-bc01-4d0a-837d-fa4c8d45cfbb tempest-VolumeQuotaClassesTest-1642285887 tempest-VolumeQuotaClassesTest-1642285887-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:57.939688 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25026cb4-bc01-4d0a-837d-fa4c8d45cfbb tempest-VolumeQuotaClassesTest-1642285887 tempest-VolumeQuotaClassesTest-1642285887-project-admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:57.940149 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25026cb4-bc01-4d0a-837d-fa4c8d45cfbb tempest-VolumeQuotaClassesTest-1642285887 tempest-VolumeQuotaClassesTest-1642285887-project-admin] PATH_INFO: `/v3/projects/3139fc3e6e48458d827bba6a2547a1c0` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:57.942343 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25026cb4-bc01-4d0a-837d-fa4c8d45cfbb tempest-VolumeQuotaClassesTest-1642285887 tempest-VolumeQuotaClassesTest-1642285887-project-admin] RBAC: Authorizing `identity:delete_project(project_id=3139fc3e6e48458d827bba6a2547a1c0)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:57.946183 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1383/2778] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:53:57 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 267 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:53:57.948008 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25026cb4-bc01-4d0a-837d-fa4c8d45cfbb tempest-VolumeQuotaClassesTest-1642285887 tempest-VolumeQuotaClassesTest-1642285887-project-admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:57.993633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=97,keystone:INSERT=1,keystone:DELETE=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:58.014378 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-25026cb4-bc01-4d0a-837d-fa4c8d45cfbb tempest-VolumeQuotaClassesTest-1642285887 tempest-VolumeQuotaClassesTest-1642285887-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3139fc3e6e48458d827bba6a2547a1c0', 'request_id': 'req-25026cb4-bc01-4d0a-837d-fa4c8d45cfbb'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:58.022741 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-25026cb4-bc01-4d0a-837d-fa4c8d45cfbb tempest-VolumeQuotaClassesTest-1642285887 tempest-VolumeQuotaClassesTest-1642285887-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3139fc3e6e48458d827bba6a2547a1c0', 'request_id': 'req-25026cb4-bc01-4d0a-837d-fa4c8d45cfbb'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:58.033317 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-25026cb4-bc01-4d0a-837d-fa4c8d45cfbb tempest-VolumeQuotaClassesTest-1642285887 tempest-VolumeQuotaClassesTest-1642285887-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3139fc3e6e48458d827bba6a2547a1c0', 'request_id': 'req-25026cb4-bc01-4d0a-837d-fa4c8d45cfbb'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:58.049611 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-25026cb4-bc01-4d0a-837d-fa4c8d45cfbb tempest-VolumeQuotaClassesTest-1642285887 tempest-VolumeQuotaClassesTest-1642285887-project-admin] The token cache is being invalidate because project 3139fc3e6e48458d827bba6a2547a1c0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:53:58.050126 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-25026cb4-bc01-4d0a-837d-fa4c8d45cfbb tempest-VolumeQuotaClassesTest-1642285887 tempest-VolumeQuotaClassesTest-1642285887-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:58.051012 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-25026cb4-bc01-4d0a-837d-fa4c8d45cfbb tempest-VolumeQuotaClassesTest-1642285887 tempest-VolumeQuotaClassesTest-1642285887-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:58.052857 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1396/2779] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:53:57 2022] DELETE /identity/v3/projects/3139fc3e6e48458d827bba6a2547a1c0 => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:53:58.398815 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38228714-730d-4afb-87ff-1c6fa8c42241 service cinder] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:58.401722 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-38228714-730d-4afb-87ff-1c6fa8c42241 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:58.521419 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:58.569489 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38228714-730d-4afb-87ff-1c6fa8c42241 service cinder] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:58.651762 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e431586-859c-4e84-b76a-a1252909c754 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:58.652874 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e431586-859c-4e84-b76a-a1252909c754 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:58.653421 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e431586-859c-4e84-b76a-a1252909c754 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:58.653910 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e431586-859c-4e84-b76a-a1252909c754 admin admin] PATH_INFO: `/v3/users/2a25ebb07ec64e5590a4a3ce3150b779` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:58.656213 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e431586-859c-4e84-b76a-a1252909c754 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2a25ebb07ec64e5590a4a3ce3150b779)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:58.660099 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e431586-859c-4e84-b76a-a1252909c754 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:58.746001 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-1e431586-859c-4e84-b76a-a1252909c754 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2a25ebb07ec64e5590a4a3ce3150b779'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:58.758653 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-1e431586-859c-4e84-b76a-a1252909c754 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2a25ebb07ec64e5590a4a3ce3150b779'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:58.759537 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-1e431586-859c-4e84-b76a-a1252909c754 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2a25ebb07ec64e5590a4a3ce3150b779'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:58.760374 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-1e431586-859c-4e84-b76a-a1252909c754 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2a25ebb07ec64e5590a4a3ce3150b779'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:58.772594 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-1e431586-859c-4e84-b76a-a1252909c754 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2a25ebb07ec64e5590a4a3ce3150b779'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:58.794319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1384/2780] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:53:58 2022] DELETE /identity/v3/users/2a25ebb07ec64e5590a4a3ce3150b779 => generated 0 bytes in 397 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:58.799366 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0181865c-d70c-47cb-b015-4a6d799abdcb None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:58.800098 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0181865c-d70c-47cb-b015-4a6d799abdcb None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:58.800098 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0181865c-d70c-47cb-b015-4a6d799abdcb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:58.821338 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-0181865c-d70c-47cb-b015-4a6d799abdcb None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:58.822628 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-0181865c-d70c-47cb-b015-4a6d799abdcb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:58.906774 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1397/2781] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:53:58.917521 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e431586-859c-4e84-b76a-a1252909c754 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:58.919026 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-1e431586-859c-4e84-b76a-a1252909c754 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:58.988023 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e431586-859c-4e84-b76a-a1252909c754 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:58.992155 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b374b33-2827-4ed9-b674-d9215c3062a9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:58.992797 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b374b33-2827-4ed9-b674-d9215c3062a9 service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:58.993087 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b374b33-2827-4ed9-b674-d9215c3062a9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:58.993364 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b374b33-2827-4ed9-b674-d9215c3062a9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:59.013338 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b374b33-2827-4ed9-b674-d9215c3062a9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:59.016365 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b374b33-2827-4ed9-b674-d9215c3062a9 service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:59.037001 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1385/2782] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:53:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:53:59.414741 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0181865c-d70c-47cb-b015-4a6d799abdcb None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:59.417200 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-0181865c-d70c-47cb-b015-4a6d799abdcb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:59.442072 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0181865c-d70c-47cb-b015-4a6d799abdcb None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:59.451842 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8527d8b-f08b-4159-bbdb-65a8b3a793dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:59.453369 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8527d8b-f08b-4159-bbdb-65a8b3a793dd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:59.454060 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8527d8b-f08b-4159-bbdb-65a8b3a793dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:59.454670 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8527d8b-f08b-4159-bbdb-65a8b3a793dd admin admin] PATH_INFO: `/v3/projects/f0223b014e5a4a6284f2babb5bbf3ace` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:59.456829 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8527d8b-f08b-4159-bbdb-65a8b3a793dd admin admin] RBAC: Authorizing `identity:delete_project(project_id=f0223b014e5a4a6284f2babb5bbf3ace)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:59.460673 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8527d8b-f08b-4159-bbdb-65a8b3a793dd admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:59.520342 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c81d27-b083-4861-8760-616d9c96b55c None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:59.521216 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c81d27-b083-4861-8760-616d9c96b55c None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:59.522599 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c81d27-b083-4861-8760-616d9c96b55c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:59.532306 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:59.550186 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-67c81d27-b083-4861-8760-616d9c96b55c None None] MFA Rules not processed for user `4d088636881e45559699462268bd0411`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:53:59.552157 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-67c81d27-b083-4861-8760-616d9c96b55c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:59.562523 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-d8527d8b-f08b-4159-bbdb-65a8b3a793dd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f0223b014e5a4a6284f2babb5bbf3ace', 'request_id': 'req-d8527d8b-f08b-4159-bbdb-65a8b3a793dd'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:59.581748 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-d8527d8b-f08b-4159-bbdb-65a8b3a793dd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f0223b014e5a4a6284f2babb5bbf3ace', 'request_id': 'req-d8527d8b-f08b-4159-bbdb-65a8b3a793dd'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:59.597746 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-d8527d8b-f08b-4159-bbdb-65a8b3a793dd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f0223b014e5a4a6284f2babb5bbf3ace', 'request_id': 'req-d8527d8b-f08b-4159-bbdb-65a8b3a793dd'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:59.623915 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-d8527d8b-f08b-4159-bbdb-65a8b3a793dd admin admin] The token cache is being invalidate because project f0223b014e5a4a6284f2babb5bbf3ace was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:53:59.624551 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-d8527d8b-f08b-4159-bbdb-65a8b3a793dd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:59.625766 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-d8527d8b-f08b-4159-bbdb-65a8b3a793dd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:59.628529 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1398/2783] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:53:59 2022] DELETE /identity/v3/projects/f0223b014e5a4a6284f2babb5bbf3ace => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:53:59.633923 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8527d8b-f08b-4159-bbdb-65a8b3a793dd admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:53:59.637078 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d8527d8b-f08b-4159-bbdb-65a8b3a793dd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:53:59.652546 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1386/2784] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:53:59 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:53:59.709928 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:53:59.742116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8527d8b-f08b-4159-bbdb-65a8b3a793dd admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:53:59.757987 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-023cf47d-5020-4812-8a56-799299bb8597 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:53:59.759209 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023cf47d-5020-4812-8a56-799299bb8597 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:53:59.760005 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023cf47d-5020-4812-8a56-799299bb8597 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:53:59.760578 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023cf47d-5020-4812-8a56-799299bb8597 admin admin] PATH_INFO: `/v3/users/129eeefcec1e4136aba36eb6425a3c3c` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:53:59.763503 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-023cf47d-5020-4812-8a56-799299bb8597 admin admin] RBAC: Authorizing `identity:delete_user(user_id=129eeefcec1e4136aba36eb6425a3c3c)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:53:59.768098 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-023cf47d-5020-4812-8a56-799299bb8597 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:53:59.838046 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-023cf47d-5020-4812-8a56-799299bb8597 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '129eeefcec1e4136aba36eb6425a3c3c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:59.847945 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-023cf47d-5020-4812-8a56-799299bb8597 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '129eeefcec1e4136aba36eb6425a3c3c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:59.848653 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-023cf47d-5020-4812-8a56-799299bb8597 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '129eeefcec1e4136aba36eb6425a3c3c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:59.849411 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-023cf47d-5020-4812-8a56-799299bb8597 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '129eeefcec1e4136aba36eb6425a3c3c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:59.859122 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-023cf47d-5020-4812-8a56-799299bb8597 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '129eeefcec1e4136aba36eb6425a3c3c'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:53:59.885673 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1399/2785] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:53:59 2022] DELETE /identity/v3/users/129eeefcec1e4136aba36eb6425a3c3c => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:54:00.309046 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c81d27-b083-4861-8760-616d9c96b55c None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:00.327846 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c81d27-b083-4861-8760-616d9c96b55c None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:00.433453 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db3d7e7-1906-437e-af1d-908c5ccc8294 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:00.438848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db3d7e7-1906-437e-af1d-908c5ccc8294 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:00.439139 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db3d7e7-1906-437e-af1d-908c5ccc8294 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:00.439680 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db3d7e7-1906-437e-af1d-908c5ccc8294 admin admin] PATH_INFO: `/v3/projects/b288dcf97b964751b4cff1cc6460b022` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:00.441126 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db3d7e7-1906-437e-af1d-908c5ccc8294 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b288dcf97b964751b4cff1cc6460b022)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:00.443835 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db3d7e7-1906-437e-af1d-908c5ccc8294 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:00.506203 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-1db3d7e7-1906-437e-af1d-908c5ccc8294 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b288dcf97b964751b4cff1cc6460b022', 'request_id': 'req-1db3d7e7-1906-437e-af1d-908c5ccc8294'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:00.515515 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-1db3d7e7-1906-437e-af1d-908c5ccc8294 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b288dcf97b964751b4cff1cc6460b022', 'request_id': 'req-1db3d7e7-1906-437e-af1d-908c5ccc8294'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:00.525671 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-1db3d7e7-1906-437e-af1d-908c5ccc8294 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b288dcf97b964751b4cff1cc6460b022', 'request_id': 'req-1db3d7e7-1906-437e-af1d-908c5ccc8294'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:00.542662 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-1db3d7e7-1906-437e-af1d-908c5ccc8294 admin admin] The token cache is being invalidate because project b288dcf97b964751b4cff1cc6460b022 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:54:00.542964 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-1db3d7e7-1906-437e-af1d-908c5ccc8294 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:00.543798 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-1db3d7e7-1906-437e-af1d-908c5ccc8294 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:00.546447 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1387/2786] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:54:00 2022] DELETE /identity/v3/projects/b288dcf97b964751b4cff1cc6460b022 => generated 0 bytes in 238 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:54:00.553242 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47970fe-82e8-42c4-af52-ea7c24300e8d None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:00.553872 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47970fe-82e8-42c4-af52-ea7c24300e8d None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:00.554351 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47970fe-82e8-42c4-af52-ea7c24300e8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:00.607846 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-a47970fe-82e8-42c4-af52-ea7c24300e8d None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:54:00.609646 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a47970fe-82e8-42c4-af52-ea7c24300e8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:00.733788 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1400/2787] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:54:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:54:00.749607 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dec1a71-535f-47a3-9ba4-c4997dccd429 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:00.750093 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dec1a71-535f-47a3-9ba4-c4997dccd429 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:00.750625 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dec1a71-535f-47a3-9ba4-c4997dccd429 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:00.789201 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-1dec1a71-535f-47a3-9ba4-c4997dccd429 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:54:00.791467 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-1dec1a71-535f-47a3-9ba4-c4997dccd429 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:00.808056 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1388/2788] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:54:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 60 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:54:00.814975 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a47970fe-82e8-42c4-af52-ea7c24300e8d None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:00.835285 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a47970fe-82e8-42c4-af52-ea7c24300e8d None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:00.844205 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c27d1f8-3d43-4a82-a521-9d6b566c7ada admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:00.845043 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c27d1f8-3d43-4a82-a521-9d6b566c7ada admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:00.845332 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c27d1f8-3d43-4a82-a521-9d6b566c7ada admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:00.845827 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c27d1f8-3d43-4a82-a521-9d6b566c7ada admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:00.847221 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c27d1f8-3d43-4a82-a521-9d6b566c7ada admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:54:00.847746 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c27d1f8-3d43-4a82-a521-9d6b566c7ada admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:00.852819 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c27d1f8-3d43-4a82-a521-9d6b566c7ada admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:00.894994 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1401/2789] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:54:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Oct 28 09:54:00.900611 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dec1a71-535f-47a3-9ba4-c4997dccd429 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:00.918965 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dec1a71-535f-47a3-9ba4-c4997dccd429 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:00.927838 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28277e47-5ebc-44c2-b257-cd92a4e8c4d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:00.928861 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28277e47-5ebc-44c2-b257-cd92a4e8c4d4 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:00.929783 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28277e47-5ebc-44c2-b257-cd92a4e8c4d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:00.929783 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28277e47-5ebc-44c2-b257-cd92a4e8c4d4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:00.931071 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28277e47-5ebc-44c2-b257-cd92a4e8c4d4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:00.935766 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28277e47-5ebc-44c2-b257-cd92a4e8c4d4 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:00.994616 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1389/2790] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:54:00 2022] POST /identity/v3/projects => generated 387 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:54:01.000760 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c27d1f8-3d43-4a82-a521-9d6b566c7ada admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:01.018865 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c27d1f8-3d43-4a82-a521-9d6b566c7ada admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:01.133189 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:54:01.155053 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-677c0d47-5ea7-4874-9492-74dae34b593b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:01.157323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677c0d47-5ea7-4874-9492-74dae34b593b admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:01.157323 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677c0d47-5ea7-4874-9492-74dae34b593b admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:01.157909 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-677c0d47-5ea7-4874-9492-74dae34b593b admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:01.158872 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-677c0d47-5ea7-4874-9492-74dae34b593b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:01.162286 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-677c0d47-5ea7-4874-9492-74dae34b593b admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:01.207164 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1402/2791] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:54:00 2022] POST /identity/v3/users => generated 294 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:54:01.209415 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28277e47-5ebc-44c2-b257-cd92a4e8c4d4 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:01.227612 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28277e47-5ebc-44c2-b257-cd92a4e8c4d4 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:01.233455 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21d04a9-b920-4664-ba92-9fede10f695d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:01.234153 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21d04a9-b920-4664-ba92-9fede10f695d admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:01.234535 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21d04a9-b920-4664-ba92-9fede10f695d admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:01.234919 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21d04a9-b920-4664-ba92-9fede10f695d admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:01.235631 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e21d04a9-b920-4664-ba92-9fede10f695d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:01.237366 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e21d04a9-b920-4664-ba92-9fede10f695d admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:01.245230 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:54:01.253967 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1390/2792] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:54:01 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:54:01.259703 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-677c0d47-5ea7-4874-9492-74dae34b593b admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:01.274136 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-677c0d47-5ea7-4874-9492-74dae34b593b admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:01.283730 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-644b26a9-c8e4-4eb9-94bd-249a0b626168 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:01.284942 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644b26a9-c8e4-4eb9-94bd-249a0b626168 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:01.285513 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644b26a9-c8e4-4eb9-94bd-249a0b626168 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:01.285999 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644b26a9-c8e4-4eb9-94bd-249a0b626168 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:01.287243 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-644b26a9-c8e4-4eb9-94bd-249a0b626168 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:01.291149 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-644b26a9-c8e4-4eb9-94bd-249a0b626168 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:01.313642 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1403/2793] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:54:01 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:54:01.319762 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21d04a9-b920-4664-ba92-9fede10f695d admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:01.342148 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21d04a9-b920-4664-ba92-9fede10f695d admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:01.350648 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a817193d-5b7f-4ebb-8b36-b88a273cff17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:01.351839 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a817193d-5b7f-4ebb-8b36-b88a273cff17 admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:01.352328 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a817193d-5b7f-4ebb-8b36-b88a273cff17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:01.352799 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a817193d-5b7f-4ebb-8b36-b88a273cff17 admin admin] PATH_INFO: `/v3/projects/4b8f62b0c70e43458b58573f39bc7063/users/f4b73dd0c746447da083e1de321c798b/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:01.383850 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a817193d-5b7f-4ebb-8b36-b88a273cff17 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4b8f62b0c70e43458b58573f39bc7063, user_id=f4b73dd0c746447da083e1de321c798b, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:01.389864 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a817193d-5b7f-4ebb-8b36-b88a273cff17 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:01.413252 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1391/2794] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:54:01 2022] PUT /identity/v3/projects/4b8f62b0c70e43458b58573f39bc7063/users/f4b73dd0c746447da083e1de321c798b/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 95 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:54:01.431918 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392e282f-3318-4cde-b930-bdb13f3420ee None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:01.432427 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392e282f-3318-4cde-b930-bdb13f3420ee None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:01.432923 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392e282f-3318-4cde-b930-bdb13f3420ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:01.473696 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-392e282f-3318-4cde-b930-bdb13f3420ee None None] MFA Rules not processed for user `f4b73dd0c746447da083e1de321c798b`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:54:01.475293 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-392e282f-3318-4cde-b930-bdb13f3420ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:01.625401 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1404/2795] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:54:01 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:54:01.631726 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a817193d-5b7f-4ebb-8b36-b88a273cff17 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:01.707435 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a817193d-5b7f-4ebb-8b36-b88a273cff17 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:01.713076 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aafbdad2-e3df-4b96-9a3d-df460cfb1cce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:01.713918 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafbdad2-e3df-4b96-9a3d-df460cfb1cce admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:01.714205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafbdad2-e3df-4b96-9a3d-df460cfb1cce admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:01.718518 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafbdad2-e3df-4b96-9a3d-df460cfb1cce admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:01.718518 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aafbdad2-e3df-4b96-9a3d-df460cfb1cce admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:01.718518 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aafbdad2-e3df-4b96-9a3d-df460cfb1cce admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:01.749652 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1392/2796] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:54:01 2022] POST /identity/v3/projects => generated 385 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:54:01.753970 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-392e282f-3318-4cde-b930-bdb13f3420ee None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:01.765882 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-392e282f-3318-4cde-b930-bdb13f3420ee None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:01.831065 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb92b762-836b-41bc-b524-c9d2625d0cf0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:01.831771 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb92b762-836b-41bc-b524-c9d2625d0cf0 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:01.832099 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb92b762-836b-41bc-b524-c9d2625d0cf0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:01.832472 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb92b762-836b-41bc-b524-c9d2625d0cf0 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:01.833212 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb92b762-836b-41bc-b524-c9d2625d0cf0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:01.835268 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb92b762-836b-41bc-b524-c9d2625d0cf0 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:01.860634 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1405/2797] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:54:01 2022] POST /identity/v3/users => generated 293 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:54:01.864914 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aafbdad2-e3df-4b96-9a3d-df460cfb1cce admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:01.876508 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aafbdad2-e3df-4b96-9a3d-df460cfb1cce admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:01.881394 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b39ab9-0828-41b2-879b-7c88b9bcf320 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:01.882069 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b39ab9-0828-41b2-879b-7c88b9bcf320 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:01.882356 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b39ab9-0828-41b2-879b-7c88b9bcf320 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:01.882661 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b39ab9-0828-41b2-879b-7c88b9bcf320 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:01.883325 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b39ab9-0828-41b2-879b-7c88b9bcf320 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:01.885044 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b39ab9-0828-41b2-879b-7c88b9bcf320 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:01.897723 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1393/2798] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:54:01 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:54:01.902655 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb92b762-836b-41bc-b524-c9d2625d0cf0 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:01.914362 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb92b762-836b-41bc-b524-c9d2625d0cf0 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:01.919374 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f941ee-7116-4848-906b-7e6b86a5f5d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:01.920007 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f941ee-7116-4848-906b-7e6b86a5f5d1 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:01.920291 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f941ee-7116-4848-906b-7e6b86a5f5d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:01.920561 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f941ee-7116-4848-906b-7e6b86a5f5d1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:01.921220 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f941ee-7116-4848-906b-7e6b86a5f5d1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:01.923180 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f941ee-7116-4848-906b-7e6b86a5f5d1 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:01.935468 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1406/2799] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:54:01 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:54:01.940476 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b39ab9-0828-41b2-879b-7c88b9bcf320 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:01.952104 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b39ab9-0828-41b2-879b-7c88b9bcf320 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:01.957353 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49dae516-fa3a-4d89-82f2-923d443e051c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:01.957939 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49dae516-fa3a-4d89-82f2-923d443e051c admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:01.958214 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49dae516-fa3a-4d89-82f2-923d443e051c admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:01.958522 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49dae516-fa3a-4d89-82f2-923d443e051c admin admin] PATH_INFO: `/v3/projects/98dc5a7278584d84ae4e1d46a2aa3132/users/c29efab964da4becb0a0845a9bb67f2a/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:01.976767 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49dae516-fa3a-4d89-82f2-923d443e051c admin admin] RBAC: Authorizing `identity:create_grant(project_id=98dc5a7278584d84ae4e1d46a2aa3132, user_id=c29efab964da4becb0a0845a9bb67f2a, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:01.979504 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49dae516-fa3a-4d89-82f2-923d443e051c admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:01.995061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1394/2800] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:54:01 2022] PUT /identity/v3/projects/98dc5a7278584d84ae4e1d46a2aa3132/users/c29efab964da4becb0a0845a9bb67f2a/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:54:02.008477 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede12232-94f0-47bf-a92c-ac31fbdbec4d None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:02.009203 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede12232-94f0-47bf-a92c-ac31fbdbec4d None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:02.009709 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ede12232-94f0-47bf-a92c-ac31fbdbec4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:02.059150 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-ede12232-94f0-47bf-a92c-ac31fbdbec4d None None] MFA Rules not processed for user `c29efab964da4becb0a0845a9bb67f2a`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:54:02.061813 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-ede12232-94f0-47bf-a92c-ac31fbdbec4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:02.111707 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:54:02.169371 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1407/2801] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:54:02 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:54:02.173128 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49dae516-fa3a-4d89-82f2-923d443e051c admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:02.257588 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49dae516-fa3a-4d89-82f2-923d443e051c admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:02.262689 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fabe2cc9-bfb9-4012-a20f-9a93711d7fd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:02.263353 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fabe2cc9-bfb9-4012-a20f-9a93711d7fd4 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:02.263678 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fabe2cc9-bfb9-4012-a20f-9a93711d7fd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:02.263942 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fabe2cc9-bfb9-4012-a20f-9a93711d7fd4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:02.264616 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fabe2cc9-bfb9-4012-a20f-9a93711d7fd4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:02.266616 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fabe2cc9-bfb9-4012-a20f-9a93711d7fd4 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:02.313905 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1395/2802] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:54:02 2022] POST /identity/v3/projects => generated 385 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:54:02.318659 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ede12232-94f0-47bf-a92c-ac31fbdbec4d None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:02.332554 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ede12232-94f0-47bf-a92c-ac31fbdbec4d None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:02.429379 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f0beb64-97d9-43e7-8c92-40e1c6fb9b6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:02.430021 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0beb64-97d9-43e7-8c92-40e1c6fb9b6f admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:02.430305 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0beb64-97d9-43e7-8c92-40e1c6fb9b6f admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:02.430618 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0beb64-97d9-43e7-8c92-40e1c6fb9b6f admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:02.431294 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f0beb64-97d9-43e7-8c92-40e1c6fb9b6f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:02.433319 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f0beb64-97d9-43e7-8c92-40e1c6fb9b6f admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:02.457443 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1408/2803] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:54:02 2022] POST /identity/v3/users => generated 299 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:54:02.463361 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fabe2cc9-bfb9-4012-a20f-9a93711d7fd4 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:02.475883 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fabe2cc9-bfb9-4012-a20f-9a93711d7fd4 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:02.481083 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7a56d2-c762-4ace-a1d2-d7e72ed246b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:02.482373 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7a56d2-c762-4ace-a1d2-d7e72ed246b8 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:02.482783 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7a56d2-c762-4ace-a1d2-d7e72ed246b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:02.483150 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7a56d2-c762-4ace-a1d2-d7e72ed246b8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:02.483920 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb7a56d2-c762-4ace-a1d2-d7e72ed246b8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:02.485827 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb7a56d2-c762-4ace-a1d2-d7e72ed246b8 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:02.493110 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:54:02.500761 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1396/2804] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:54:02 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:54:02.506332 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f0beb64-97d9-43e7-8c92-40e1c6fb9b6f admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:02.517755 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f0beb64-97d9-43e7-8c92-40e1c6fb9b6f admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:02.525560 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a15310c-d586-4527-87b7-c911d4b86523 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:02.526785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a15310c-d586-4527-87b7-c911d4b86523 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:02.527298 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a15310c-d586-4527-87b7-c911d4b86523 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:02.527785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a15310c-d586-4527-87b7-c911d4b86523 admin admin] PATH_INFO: `/v3/projects/8f510189230a40c1afd29260a7793e88/users/b0c0f8c365644171a51eccc1c9dc0814/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:02.555903 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a15310c-d586-4527-87b7-c911d4b86523 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8f510189230a40c1afd29260a7793e88, user_id=b0c0f8c365644171a51eccc1c9dc0814, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:02.560839 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a15310c-d586-4527-87b7-c911d4b86523 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:02.582067 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1409/2805] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:54:02 2022] PUT /identity/v3/projects/8f510189230a40c1afd29260a7793e88/users/b0c0f8c365644171a51eccc1c9dc0814/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:54:02.586796 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7a56d2-c762-4ace-a1d2-d7e72ed246b8 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:02.598342 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7a56d2-c762-4ace-a1d2-d7e72ed246b8 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:02.667198 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e8491b-629a-4e03-bcad-8d3df5a18284 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:02.668066 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e8491b-629a-4e03-bcad-8d3df5a18284 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:02.668228 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e8491b-629a-4e03-bcad-8d3df5a18284 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:02.668491 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e8491b-629a-4e03-bcad-8d3df5a18284 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:02.669857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e8491b-629a-4e03-bcad-8d3df5a18284 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:02.671577 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e8491b-629a-4e03-bcad-8d3df5a18284 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:02.690454 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1397/2806] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:54:02 2022] GET /identity/v3/roles => generated 1420 bytes in 103 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:54:02.693962 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a15310c-d586-4527-87b7-c911d4b86523 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:02.708117 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a15310c-d586-4527-87b7-c911d4b86523 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:02.717783 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa2c69eb-0399-4f63-a451-c71af83d1cec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:02.719028 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2c69eb-0399-4f63-a451-c71af83d1cec admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:02.719526 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2c69eb-0399-4f63-a451-c71af83d1cec admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:02.720004 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2c69eb-0399-4f63-a451-c71af83d1cec admin admin] PATH_INFO: `/v3/projects/8f510189230a40c1afd29260a7793e88/users/b0c0f8c365644171a51eccc1c9dc0814/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:02.724116 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa2c69eb-0399-4f63-a451-c71af83d1cec admin admin] RBAC: Authorizing `identity:create_grant(project_id=8f510189230a40c1afd29260a7793e88, user_id=b0c0f8c365644171a51eccc1c9dc0814, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:02.729611 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa2c69eb-0399-4f63-a451-c71af83d1cec admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:02.751728 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1410/2807] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:54:02 2022] PUT /identity/v3/projects/8f510189230a40c1afd29260a7793e88/users/b0c0f8c365644171a51eccc1c9dc0814/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:54:02.772294 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b0d6c2-1a72-469c-a0dc-06e709099886 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:02.772894 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b0d6c2-1a72-469c-a0dc-06e709099886 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:02.773149 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b0d6c2-1a72-469c-a0dc-06e709099886 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:02.830010 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-c0b0d6c2-1a72-469c-a0dc-06e709099886 None None] MFA Rules not processed for user `b0c0f8c365644171a51eccc1c9dc0814`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:54:02.832231 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-c0b0d6c2-1a72-469c-a0dc-06e709099886 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:02.983111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1398/2808] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:54:02 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:54:02.994960 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa2c69eb-0399-4f63-a451-c71af83d1cec admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:02.997581 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-aa2c69eb-0399-4f63-a451-c71af83d1cec admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:03.075711 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa2c69eb-0399-4f63-a451-c71af83d1cec admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:03.080079 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79bd9982-0fe1-4d58-a401-eeb0014d056a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:03.081129 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79bd9982-0fe1-4d58-a401-eeb0014d056a service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:03.081623 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79bd9982-0fe1-4d58-a401-eeb0014d056a service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:03.082108 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79bd9982-0fe1-4d58-a401-eeb0014d056a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:03.095021 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79bd9982-0fe1-4d58-a401-eeb0014d056a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:03.096475 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79bd9982-0fe1-4d58-a401-eeb0014d056a service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:03.109138 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1411/2809] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:54:02 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:54:03.985089 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b0d6c2-1a72-469c-a0dc-06e709099886 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:04.000429 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b0d6c2-1a72-469c-a0dc-06e709099886 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:04.007122 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e3b5764-c199-4e3c-b0cd-0ab75fcd7834 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:04.008171 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3b5764-c199-4e3c-b0cd-0ab75fcd7834 service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:04.008558 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3b5764-c199-4e3c-b0cd-0ab75fcd7834 service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:04.009293 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3b5764-c199-4e3c-b0cd-0ab75fcd7834 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:04.127909 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e3b5764-c199-4e3c-b0cd-0ab75fcd7834 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:04.130072 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e3b5764-c199-4e3c-b0cd-0ab75fcd7834 service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:04.147389 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1399/2810] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:54:03 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 163 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:54:08.733838 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79bd9982-0fe1-4d58-a401-eeb0014d056a service cinder] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:08.745384 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79bd9982-0fe1-4d58-a401-eeb0014d056a service cinder] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:08.750095 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a02e83f-23ad-4b4c-b640-2ad62c71e595 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:08.750791 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a02e83f-23ad-4b4c-b640-2ad62c71e595 service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:08.751075 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a02e83f-23ad-4b4c-b640-2ad62c71e595 service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:08.751508 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a02e83f-23ad-4b4c-b640-2ad62c71e595 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:08.777172 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:54:08.831387 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a02e83f-23ad-4b4c-b640-2ad62c71e595 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:08.832906 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a02e83f-23ad-4b4c-b640-2ad62c71e595 service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:08.843695 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1412/2811] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:54:08 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:54:09.969024 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c81ac98-292a-4516-8394-18b43f11ae98 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:09.969671 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c81ac98-292a-4516-8394-18b43f11ae98 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:09.970033 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c81ac98-292a-4516-8394-18b43f11ae98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:10.000751 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-9c81ac98-292a-4516-8394-18b43f11ae98 None None] MFA Rules not processed for user `a3c6fa6f402143b598108c6cb2aedf78`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:54:10.002331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-9c81ac98-292a-4516-8394-18b43f11ae98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:10.019800 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:54:10.110455 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1400/2812] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:54:09 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:54:11.603851 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a02e83f-23ad-4b4c-b640-2ad62c71e595 service cinder] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:11.701567 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a02e83f-23ad-4b4c-b640-2ad62c71e595 service cinder] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:11.706661 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02bf3129-a364-42b6-b77d-1d9c78da5313 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:11.707384 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02bf3129-a364-42b6-b77d-1d9c78da5313 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:11.707662 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02bf3129-a364-42b6-b77d-1d9c78da5313 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:11.708033 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02bf3129-a364-42b6-b77d-1d9c78da5313 admin admin] PATH_INFO: `/v3/users/f4b73dd0c746447da083e1de321c798b` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:11.709328 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02bf3129-a364-42b6-b77d-1d9c78da5313 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f4b73dd0c746447da083e1de321c798b)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:11.711496 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02bf3129-a364-42b6-b77d-1d9c78da5313 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:11.778808 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-02bf3129-a364-42b6-b77d-1d9c78da5313 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f4b73dd0c746447da083e1de321c798b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:11.790289 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-02bf3129-a364-42b6-b77d-1d9c78da5313 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f4b73dd0c746447da083e1de321c798b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:11.791328 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-02bf3129-a364-42b6-b77d-1d9c78da5313 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f4b73dd0c746447da083e1de321c798b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:11.792410 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-02bf3129-a364-42b6-b77d-1d9c78da5313 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f4b73dd0c746447da083e1de321c798b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:11.808498 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-02bf3129-a364-42b6-b77d-1d9c78da5313 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f4b73dd0c746447da083e1de321c798b'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:11.839372 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1413/2813] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:54:11 2022] DELETE /identity/v3/users/f4b73dd0c746447da083e1de321c798b => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:54:11.846355 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639bdd34-3ba6-4cc5-b0ac-44dddde4e2e2 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:11.847263 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639bdd34-3ba6-4cc5-b0ac-44dddde4e2e2 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:11.847434 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-639bdd34-3ba6-4cc5-b0ac-44dddde4e2e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:11.874217 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-639bdd34-3ba6-4cc5-b0ac-44dddde4e2e2 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:54:11.876436 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-639bdd34-3ba6-4cc5-b0ac-44dddde4e2e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:11.964646 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1401/2814] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:54:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:54:11.978994 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02bf3129-a364-42b6-b77d-1d9c78da5313 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:11.980820 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-02bf3129-a364-42b6-b77d-1d9c78da5313 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:12.055915 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:54:12.078982 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02bf3129-a364-42b6-b77d-1d9c78da5313 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:12.083086 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36055a9d-eec2-48a3-aa82-c0e58f69ba31 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:12.083736 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36055a9d-eec2-48a3-aa82-c0e58f69ba31 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:12.084020 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36055a9d-eec2-48a3-aa82-c0e58f69ba31 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:12.084303 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36055a9d-eec2-48a3-aa82-c0e58f69ba31 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:12.095741 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36055a9d-eec2-48a3-aa82-c0e58f69ba31 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:12.097583 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36055a9d-eec2-48a3-aa82-c0e58f69ba31 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:12.109865 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1414/2815] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:54:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:54:12.492974 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-639bdd34-3ba6-4cc5-b0ac-44dddde4e2e2 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:12.494774 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-639bdd34-3ba6-4cc5-b0ac-44dddde4e2e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:12.513908 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-639bdd34-3ba6-4cc5-b0ac-44dddde4e2e2 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:12.523986 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c885a1f6-ac21-4f89-93a1-dabde995c812 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:12.525159 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c885a1f6-ac21-4f89-93a1-dabde995c812 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:12.527595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c885a1f6-ac21-4f89-93a1-dabde995c812 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:12.528116 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c885a1f6-ac21-4f89-93a1-dabde995c812 admin admin] PATH_INFO: `/v3/projects/4b8f62b0c70e43458b58573f39bc7063` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:12.530064 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c885a1f6-ac21-4f89-93a1-dabde995c812 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4b8f62b0c70e43458b58573f39bc7063)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:12.532329 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c885a1f6-ac21-4f89-93a1-dabde995c812 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:12.688391 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-c885a1f6-ac21-4f89-93a1-dabde995c812 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4b8f62b0c70e43458b58573f39bc7063', 'request_id': 'req-c885a1f6-ac21-4f89-93a1-dabde995c812'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:12.706652 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-c885a1f6-ac21-4f89-93a1-dabde995c812 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4b8f62b0c70e43458b58573f39bc7063', 'request_id': 'req-c885a1f6-ac21-4f89-93a1-dabde995c812'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:12.726612 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-c885a1f6-ac21-4f89-93a1-dabde995c812 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4b8f62b0c70e43458b58573f39bc7063', 'request_id': 'req-c885a1f6-ac21-4f89-93a1-dabde995c812'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:12.747673 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-c885a1f6-ac21-4f89-93a1-dabde995c812 admin admin] The token cache is being invalidate because project 4b8f62b0c70e43458b58573f39bc7063 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:54:12.748151 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-c885a1f6-ac21-4f89-93a1-dabde995c812 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:12.749740 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-c885a1f6-ac21-4f89-93a1-dabde995c812 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:12.757034 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1402/2816] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:54:12 2022] DELETE /identity/v3/projects/4b8f62b0c70e43458b58573f39bc7063 => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:54:12.758356 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36055a9d-eec2-48a3-aa82-c0e58f69ba31 service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:12.783361 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36055a9d-eec2-48a3-aa82-c0e58f69ba31 service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:12.916298 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17133cc6-6f2e-4afe-8984-49861aba9c4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:12.917389 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17133cc6-6f2e-4afe-8984-49861aba9c4e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:12.919391 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17133cc6-6f2e-4afe-8984-49861aba9c4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:12.920069 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17133cc6-6f2e-4afe-8984-49861aba9c4e admin admin] PATH_INFO: `/v3/users/c29efab964da4becb0a0845a9bb67f2a` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:12.921779 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17133cc6-6f2e-4afe-8984-49861aba9c4e admin admin] RBAC: Authorizing `identity:delete_user(user_id=c29efab964da4becb0a0845a9bb67f2a)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:12.924758 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17133cc6-6f2e-4afe-8984-49861aba9c4e admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:12.994811 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-17133cc6-6f2e-4afe-8984-49861aba9c4e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c29efab964da4becb0a0845a9bb67f2a'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:13.006843 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-17133cc6-6f2e-4afe-8984-49861aba9c4e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c29efab964da4becb0a0845a9bb67f2a'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:13.007810 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-17133cc6-6f2e-4afe-8984-49861aba9c4e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c29efab964da4becb0a0845a9bb67f2a'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:13.008817 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-17133cc6-6f2e-4afe-8984-49861aba9c4e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c29efab964da4becb0a0845a9bb67f2a'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:13.018159 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-17133cc6-6f2e-4afe-8984-49861aba9c4e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c29efab964da4becb0a0845a9bb67f2a'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:13.040231 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1415/2817] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:54:12 2022] DELETE /identity/v3/users/c29efab964da4becb0a0845a9bb67f2a => generated 0 bytes in 283 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:54:13.384023 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c885a1f6-ac21-4f89-93a1-dabde995c812 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:13.386318 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-c885a1f6-ac21-4f89-93a1-dabde995c812 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:13.412132 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c885a1f6-ac21-4f89-93a1-dabde995c812 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:13.507260 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:54:13.537306 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bac1ef0-7781-431b-95fd-867619a8d19e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:13.538381 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bac1ef0-7781-431b-95fd-867619a8d19e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:13.538882 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bac1ef0-7781-431b-95fd-867619a8d19e admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:13.539566 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bac1ef0-7781-431b-95fd-867619a8d19e admin admin] PATH_INFO: `/v3/projects/98dc5a7278584d84ae4e1d46a2aa3132` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:13.541937 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bac1ef0-7781-431b-95fd-867619a8d19e admin admin] RBAC: Authorizing `identity:delete_project(project_id=98dc5a7278584d84ae4e1d46a2aa3132)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:13.545957 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bac1ef0-7781-431b-95fd-867619a8d19e admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:13.636074 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-5bac1ef0-7781-431b-95fd-867619a8d19e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '98dc5a7278584d84ae4e1d46a2aa3132', 'request_id': 'req-5bac1ef0-7781-431b-95fd-867619a8d19e'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:13.644437 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-5bac1ef0-7781-431b-95fd-867619a8d19e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '98dc5a7278584d84ae4e1d46a2aa3132', 'request_id': 'req-5bac1ef0-7781-431b-95fd-867619a8d19e'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:13.656383 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-5bac1ef0-7781-431b-95fd-867619a8d19e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '98dc5a7278584d84ae4e1d46a2aa3132', 'request_id': 'req-5bac1ef0-7781-431b-95fd-867619a8d19e'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:13.677739 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-5bac1ef0-7781-431b-95fd-867619a8d19e admin admin] The token cache is being invalidate because project 98dc5a7278584d84ae4e1d46a2aa3132 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:54:13.678503 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-5bac1ef0-7781-431b-95fd-867619a8d19e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:13.679839 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-5bac1ef0-7781-431b-95fd-867619a8d19e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:13.682473 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1403/2818] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:54:13 2022] DELETE /identity/v3/projects/98dc5a7278584d84ae4e1d46a2aa3132 => generated 0 bytes in 300 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:54:13.689206 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17133cc6-6f2e-4afe-8984-49861aba9c4e admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:13.691299 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-17133cc6-6f2e-4afe-8984-49861aba9c4e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:13.711564 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17133cc6-6f2e-4afe-8984-49861aba9c4e admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:13.803781 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f384e9-7cf1-4db7-82c6-69a2010e0dad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:13.804816 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f384e9-7cf1-4db7-82c6-69a2010e0dad admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:13.805158 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f384e9-7cf1-4db7-82c6-69a2010e0dad admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:13.805611 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f384e9-7cf1-4db7-82c6-69a2010e0dad admin admin] PATH_INFO: `/v3/users/b0c0f8c365644171a51eccc1c9dc0814` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:13.807348 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15f384e9-7cf1-4db7-82c6-69a2010e0dad admin admin] RBAC: Authorizing `identity:delete_user(user_id=b0c0f8c365644171a51eccc1c9dc0814)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:13.811574 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15f384e9-7cf1-4db7-82c6-69a2010e0dad admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:13.815241 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:54:13.909401 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-15f384e9-7cf1-4db7-82c6-69a2010e0dad admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b0c0f8c365644171a51eccc1c9dc0814'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:13.922290 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-15f384e9-7cf1-4db7-82c6-69a2010e0dad admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b0c0f8c365644171a51eccc1c9dc0814'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:13.923679 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-15f384e9-7cf1-4db7-82c6-69a2010e0dad admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b0c0f8c365644171a51eccc1c9dc0814'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:13.924665 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-15f384e9-7cf1-4db7-82c6-69a2010e0dad admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b0c0f8c365644171a51eccc1c9dc0814'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:13.936868 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-15f384e9-7cf1-4db7-82c6-69a2010e0dad admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b0c0f8c365644171a51eccc1c9dc0814'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:13.960780 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1416/2819] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:54:13 2022] DELETE /identity/v3/users/b0c0f8c365644171a51eccc1c9dc0814 => generated 0 bytes in 272 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:54:14.385649 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bac1ef0-7781-431b-95fd-867619a8d19e admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:14.389596 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-5bac1ef0-7781-431b-95fd-867619a8d19e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:14.414100 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bac1ef0-7781-431b-95fd-867619a8d19e admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:14.504775 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e84acaf7-c082-4569-b702-2021c349f388 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:14.505803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84acaf7-c082-4569-b702-2021c349f388 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:14.506218 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84acaf7-c082-4569-b702-2021c349f388 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:14.506647 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84acaf7-c082-4569-b702-2021c349f388 admin admin] PATH_INFO: `/v3/projects/8f510189230a40c1afd29260a7793e88` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:14.508057 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e84acaf7-c082-4569-b702-2021c349f388 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8f510189230a40c1afd29260a7793e88)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:14.510120 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e84acaf7-c082-4569-b702-2021c349f388 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:14.627318 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-e84acaf7-c082-4569-b702-2021c349f388 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8f510189230a40c1afd29260a7793e88', 'request_id': 'req-e84acaf7-c082-4569-b702-2021c349f388'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:14.645234 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-e84acaf7-c082-4569-b702-2021c349f388 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8f510189230a40c1afd29260a7793e88', 'request_id': 'req-e84acaf7-c082-4569-b702-2021c349f388'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:14.664389 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-e84acaf7-c082-4569-b702-2021c349f388 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8f510189230a40c1afd29260a7793e88', 'request_id': 'req-e84acaf7-c082-4569-b702-2021c349f388'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:14.704187 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-e84acaf7-c082-4569-b702-2021c349f388 admin admin] The token cache is being invalidate because project 8f510189230a40c1afd29260a7793e88 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:54:14.704716 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-e84acaf7-c082-4569-b702-2021c349f388 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:14.705868 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-e84acaf7-c082-4569-b702-2021c349f388 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:14.708964 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1404/2820] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:54:14 2022] DELETE /identity/v3/projects/8f510189230a40c1afd29260a7793e88 => generated 0 bytes in 324 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:54:14.721098 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ba70aa-59f3-4904-9439-61599c4ddf23 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:14.721421 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ba70aa-59f3-4904-9439-61599c4ddf23 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:14.721644 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ba70aa-59f3-4904-9439-61599c4ddf23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:14.744790 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-a8ba70aa-59f3-4904-9439-61599c4ddf23 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:54:14.746059 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a8ba70aa-59f3-4904-9439-61599c4ddf23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:14.845934 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1417/2821] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:54:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:54:14.857397 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a255fa8-e99a-43d3-b359-678547362aac None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:14.857906 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a255fa8-e99a-43d3-b359-678547362aac None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:14.858312 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a255fa8-e99a-43d3-b359-678547362aac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:14.889000 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-0a255fa8-e99a-43d3-b359-678547362aac None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:54:14.891011 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-0a255fa8-e99a-43d3-b359-678547362aac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:14.903904 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1405/2822] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:54:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:54:14.910601 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ba70aa-59f3-4904-9439-61599c4ddf23 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:14.931073 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ba70aa-59f3-4904-9439-61599c4ddf23 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:14.941816 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99822081-6cf0-4c1c-a12e-204a874a1c1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:14.942729 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99822081-6cf0-4c1c-a12e-204a874a1c1f admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:14.943255 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99822081-6cf0-4c1c-a12e-204a874a1c1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:14.944349 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99822081-6cf0-4c1c-a12e-204a874a1c1f admin admin] PATH_INFO: `/v3/domains` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:14.945007 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99822081-6cf0-4c1c-a12e-204a874a1c1f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81577) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:54:14.945646 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99822081-6cf0-4c1c-a12e-204a874a1c1f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:14.959103 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99822081-6cf0-4c1c-a12e-204a874a1c1f admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:14.989801 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1418/2823] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:54:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:54:14.995821 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a255fa8-e99a-43d3-b359-678547362aac None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:15.005023 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:54:15.024078 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a255fa8-e99a-43d3-b359-678547362aac None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:15.041667 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c6061a-ecc7-4854-95bc-5665171a9aa0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:15.042908 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c6061a-ecc7-4854-95bc-5665171a9aa0 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:15.043238 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c6061a-ecc7-4854-95bc-5665171a9aa0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:15.044124 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c6061a-ecc7-4854-95bc-5665171a9aa0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:15.045303 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c6061a-ecc7-4854-95bc-5665171a9aa0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:15.048825 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c6061a-ecc7-4854-95bc-5665171a9aa0 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:15.115625 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1406/2824] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:54:14 2022] POST /identity/v3/projects => generated 365 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:54:15.122330 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99822081-6cf0-4c1c-a12e-204a874a1c1f admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:15.136699 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99822081-6cf0-4c1c-a12e-204a874a1c1f admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:15.260032 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81b6396c-0fbb-41d4-a477-c1f338b42af6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:15.261097 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b6396c-0fbb-41d4-a477-c1f338b42af6 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:15.262476 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b6396c-0fbb-41d4-a477-c1f338b42af6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:15.263705 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b6396c-0fbb-41d4-a477-c1f338b42af6 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:15.265021 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81b6396c-0fbb-41d4-a477-c1f338b42af6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:15.268777 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81b6396c-0fbb-41d4-a477-c1f338b42af6 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:15.292871 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:54:15.308864 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1419/2825] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:54:15 2022] POST /identity/v3/users => generated 283 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:54:15.311604 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c6061a-ecc7-4854-95bc-5665171a9aa0 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:15.324022 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c6061a-ecc7-4854-95bc-5665171a9aa0 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:15.329894 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b969998f-4f5d-4ef0-b350-db94c58f4df9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:15.330664 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b969998f-4f5d-4ef0-b350-db94c58f4df9 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:15.331166 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b969998f-4f5d-4ef0-b350-db94c58f4df9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:15.331496 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b969998f-4f5d-4ef0-b350-db94c58f4df9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:15.332248 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b969998f-4f5d-4ef0-b350-db94c58f4df9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:15.334070 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b969998f-4f5d-4ef0-b350-db94c58f4df9 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:15.347212 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1407/2826] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:54:15 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:54:15.351911 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81b6396c-0fbb-41d4-a477-c1f338b42af6 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:15.363832 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81b6396c-0fbb-41d4-a477-c1f338b42af6 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:15.368748 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f80070c-3a2b-408e-ab10-9b4636110194 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:15.369378 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f80070c-3a2b-408e-ab10-9b4636110194 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:15.369658 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f80070c-3a2b-408e-ab10-9b4636110194 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:15.369931 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f80070c-3a2b-408e-ab10-9b4636110194 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:15.370621 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f80070c-3a2b-408e-ab10-9b4636110194 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:15.372551 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f80070c-3a2b-408e-ab10-9b4636110194 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:15.388503 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1420/2827] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:54:15 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:54:15.390166 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b969998f-4f5d-4ef0-b350-db94c58f4df9 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:15.402572 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b969998f-4f5d-4ef0-b350-db94c58f4df9 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:15.408341 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7c7d77-c083-4d3d-9afe-a8e510a2220d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:15.409181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7c7d77-c083-4d3d-9afe-a8e510a2220d admin admin] REQUEST_METHOD: `PUT` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:15.409549 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7c7d77-c083-4d3d-9afe-a8e510a2220d admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:15.409893 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7c7d77-c083-4d3d-9afe-a8e510a2220d admin admin] PATH_INFO: `/v3/projects/a68cbdfd379248ed9145fdd7a70a17a7/users/8a1c79ab449742c3b1aaa6639b7ceaf5/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:15.431068 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7c7d77-c083-4d3d-9afe-a8e510a2220d admin admin] RBAC: Authorizing `identity:create_grant(project_id=a68cbdfd379248ed9145fdd7a70a17a7, user_id=8a1c79ab449742c3b1aaa6639b7ceaf5, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:15.434158 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7c7d77-c083-4d3d-9afe-a8e510a2220d admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:15.450278 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1408/2828] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:54:15 2022] PUT /identity/v3/projects/a68cbdfd379248ed9145fdd7a70a17a7/users/8a1c79ab449742c3b1aaa6639b7ceaf5/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:54:15.465196 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ace4c8a-a5f4-4ae5-8172-9b9940c93851 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:15.465196 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ace4c8a-a5f4-4ae5-8172-9b9940c93851 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:15.465196 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ace4c8a-a5f4-4ae5-8172-9b9940c93851 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:15.501941 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-1ace4c8a-a5f4-4ae5-8172-9b9940c93851 None None] MFA Rules not processed for user `8a1c79ab449742c3b1aaa6639b7ceaf5`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:54:15.503576 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-1ace4c8a-a5f4-4ae5-8172-9b9940c93851 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:15.607175 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1421/2829] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:54:15 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:54:15.612778 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7c7d77-c083-4d3d-9afe-a8e510a2220d admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:15.701181 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7c7d77-c083-4d3d-9afe-a8e510a2220d admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:15.711987 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a6a15d-bdc4-4dff-a129-1c009ff4e085 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:15.712955 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a6a15d-bdc4-4dff-a129-1c009ff4e085 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:15.713362 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a6a15d-bdc4-4dff-a129-1c009ff4e085 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:15.713701 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a6a15d-bdc4-4dff-a129-1c009ff4e085 admin admin] PATH_INFO: `/v3/projects` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:15.714552 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68a6a15d-bdc4-4dff-a129-1c009ff4e085 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:15.717779 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68a6a15d-bdc4-4dff-a129-1c009ff4e085 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:15.751042 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1409/2830] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:54:15 2022] POST /identity/v3/projects => generated 367 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:54:15.757330 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ace4c8a-a5f4-4ae5-8172-9b9940c93851 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:15.770144 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ace4c8a-a5f4-4ae5-8172-9b9940c93851 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:15.839243 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb68d86c-4337-437c-8233-a95686dacfa3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:15.840002 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb68d86c-4337-437c-8233-a95686dacfa3 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:15.841110 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb68d86c-4337-437c-8233-a95686dacfa3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:15.841397 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb68d86c-4337-437c-8233-a95686dacfa3 admin admin] PATH_INFO: `/v3/users` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:15.842415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb68d86c-4337-437c-8233-a95686dacfa3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:15.845534 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb68d86c-4337-437c-8233-a95686dacfa3 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:15.873523 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1422/2831] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:54:15 2022] POST /identity/v3/users => generated 290 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:54:15.879261 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a6a15d-bdc4-4dff-a129-1c009ff4e085 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:15.892547 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a6a15d-bdc4-4dff-a129-1c009ff4e085 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:15.899044 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd56ff8d-f4ca-4811-8cc4-0474632c57e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:15.899877 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd56ff8d-f4ca-4811-8cc4-0474632c57e8 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:15.900252 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd56ff8d-f4ca-4811-8cc4-0474632c57e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:15.900632 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd56ff8d-f4ca-4811-8cc4-0474632c57e8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:15.901496 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd56ff8d-f4ca-4811-8cc4-0474632c57e8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:15.903346 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd56ff8d-f4ca-4811-8cc4-0474632c57e8 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:15.917278 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1410/2832] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:54:15 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:54:15.922530 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb68d86c-4337-437c-8233-a95686dacfa3 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:15.935214 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb68d86c-4337-437c-8233-a95686dacfa3 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:15.940131 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23797416-584f-47f7-a319-8d7dae861a35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:15.940811 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23797416-584f-47f7-a319-8d7dae861a35 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:15.941090 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23797416-584f-47f7-a319-8d7dae861a35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:15.941363 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23797416-584f-47f7-a319-8d7dae861a35 admin admin] PATH_INFO: `/v3/projects/f64b7401eac942ecb30c28c81076ff60/users/559463366dae442cb501816ceb58b84d/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:15.961218 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23797416-584f-47f7-a319-8d7dae861a35 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f64b7401eac942ecb30c28c81076ff60, user_id=559463366dae442cb501816ceb58b84d, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:15.963959 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23797416-584f-47f7-a319-8d7dae861a35 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:15.986370 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1423/2833] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:54:15 2022] PUT /identity/v3/projects/f64b7401eac942ecb30c28c81076ff60/users/559463366dae442cb501816ceb58b84d/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:54:15.990665 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd56ff8d-f4ca-4811-8cc4-0474632c57e8 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:16.004327 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd56ff8d-f4ca-4811-8cc4-0474632c57e8 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:16.089331 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ccb2501-d713-4b23-99de-23371a7c7b76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:16.090894 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccb2501-d713-4b23-99de-23371a7c7b76 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:16.091368 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccb2501-d713-4b23-99de-23371a7c7b76 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:16.091847 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ccb2501-d713-4b23-99de-23371a7c7b76 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:16.093179 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ccb2501-d713-4b23-99de-23371a7c7b76 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:16.096338 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ccb2501-d713-4b23-99de-23371a7c7b76 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:16.118318 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1411/2834] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:54:15 2022] GET /identity/v3/roles => generated 1420 bytes in 128 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:54:16.123002 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23797416-584f-47f7-a319-8d7dae861a35 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:16.135057 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23797416-584f-47f7-a319-8d7dae861a35 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:16.140480 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0221f152-4afa-4523-a726-a4be03a96110 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:16.141224 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0221f152-4afa-4523-a726-a4be03a96110 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:16.141639 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0221f152-4afa-4523-a726-a4be03a96110 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:16.141921 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0221f152-4afa-4523-a726-a4be03a96110 admin admin] PATH_INFO: `/v3/projects/f64b7401eac942ecb30c28c81076ff60/users/559463366dae442cb501816ceb58b84d/roles/e13d798a98ea4c3dbb87a236ec683d32` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:16.144115 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0221f152-4afa-4523-a726-a4be03a96110 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f64b7401eac942ecb30c28c81076ff60, user_id=559463366dae442cb501816ceb58b84d, role_id=e13d798a98ea4c3dbb87a236ec683d32)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:16.146875 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0221f152-4afa-4523-a726-a4be03a96110 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:16.159435 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1424/2835] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:54:16 2022] PUT /identity/v3/projects/f64b7401eac942ecb30c28c81076ff60/users/559463366dae442cb501816ceb58b84d/roles/e13d798a98ea4c3dbb87a236ec683d32 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:54:16.229354 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de3a0e1f-cdc1-4c5f-b049-1b1eaaf59801 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:16.229522 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de3a0e1f-cdc1-4c5f-b049-1b1eaaf59801 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:16.229802 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de3a0e1f-cdc1-4c5f-b049-1b1eaaf59801 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:16.265081 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-de3a0e1f-cdc1-4c5f-b049-1b1eaaf59801 None None] MFA Rules not processed for user `559463366dae442cb501816ceb58b84d`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:54:16.266313 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-de3a0e1f-cdc1-4c5f-b049-1b1eaaf59801 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:16.270653 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:54:16.369497 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1412/2836] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:54:16 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:54:16.388351 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0221f152-4afa-4523-a726-a4be03a96110 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:16.391132 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-0221f152-4afa-4523-a726-a4be03a96110 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:16.452734 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:54:16.500328 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0221f152-4afa-4523-a726-a4be03a96110 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:16.505542 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-843caf2f-83e2-44ca-92e3-d8c7d44f5b59 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:16.506450 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843caf2f-83e2-44ca-92e3-d8c7d44f5b59 service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:16.506886 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843caf2f-83e2-44ca-92e3-d8c7d44f5b59 service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:16.507255 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843caf2f-83e2-44ca-92e3-d8c7d44f5b59 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:16.524379 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-843caf2f-83e2-44ca-92e3-d8c7d44f5b59 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:16.527698 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-843caf2f-83e2-44ca-92e3-d8c7d44f5b59 service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:16.549952 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1425/2837] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:54:16 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 163 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:54:16.664666 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de3a0e1f-cdc1-4c5f-b049-1b1eaaf59801 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:16.676120 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de3a0e1f-cdc1-4c5f-b049-1b1eaaf59801 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:16.680081 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01cffdd2-4c4c-49b0-ac9f-913066ae3877 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:16.680720 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cffdd2-4c4c-49b0-ac9f-913066ae3877 service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:16.681007 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cffdd2-4c4c-49b0-ac9f-913066ae3877 service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:16.681280 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cffdd2-4c4c-49b0-ac9f-913066ae3877 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:16.761613 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01cffdd2-4c4c-49b0-ac9f-913066ae3877 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:16.765253 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01cffdd2-4c4c-49b0-ac9f-913066ae3877 service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:16.780557 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1413/2838] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:54:16 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:54:21.917011 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914e6062-ff64-48d7-bea3-e9b9c4bf573d None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:21.917712 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914e6062-ff64-48d7-bea3-e9b9c4bf573d None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:21.917712 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-914e6062-ff64-48d7-bea3-e9b9c4bf573d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:21.953719 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-914e6062-ff64-48d7-bea3-e9b9c4bf573d None None] MFA Rules not processed for user `97c8877c74194a6bbbf25b3c8f5b7dbd`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:54:21.955088 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-914e6062-ff64-48d7-bea3-e9b9c4bf573d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:22.031837 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1426/2839] 10.222.0.25 () {58 vars in 1009 bytes} [Fri Oct 28 09:54:21 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:54:22.084665 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01cffdd2-4c4c-49b0-ac9f-913066ae3877 service cinder] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:22.086284 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-01cffdd2-4c4c-49b0-ac9f-913066ae3877 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:22.169293 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01cffdd2-4c4c-49b0-ac9f-913066ae3877 service cinder] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:22.175026 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b452681c-37d2-4ae9-8382-69824127d8e7 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c6cffe0d56e45c4b8fb54fbe5df6094', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:22.175672 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b452681c-37d2-4ae9-8382-69824127d8e7 service swift] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:22.175972 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b452681c-37d2-4ae9-8382-69824127d8e7 service swift] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:22.176261 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b452681c-37d2-4ae9-8382-69824127d8e7 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:22.187470 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b452681c-37d2-4ae9-8382-69824127d8e7 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:22.189193 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b452681c-37d2-4ae9-8382-69824127d8e7 service swift] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:22.199262 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1414/2840] 10.222.0.25 () {60 vars in 1384 bytes} [Fri Oct 28 09:54:22 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 28 09:54:22.589704 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914e6062-ff64-48d7-bea3-e9b9c4bf573d None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:22.691805 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-914e6062-ff64-48d7-bea3-e9b9c4bf573d None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:22.699152 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa665cd8-26e8-4a5e-b3b7-33406e2a9eb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:22.700878 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa665cd8-26e8-4a5e-b3b7-33406e2a9eb8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:22.701306 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa665cd8-26e8-4a5e-b3b7-33406e2a9eb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:22.701691 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa665cd8-26e8-4a5e-b3b7-33406e2a9eb8 admin admin] PATH_INFO: `/v3/users/8a1c79ab449742c3b1aaa6639b7ceaf5` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:22.703134 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa665cd8-26e8-4a5e-b3b7-33406e2a9eb8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8a1c79ab449742c3b1aaa6639b7ceaf5)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:22.706276 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa665cd8-26e8-4a5e-b3b7-33406e2a9eb8 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:22.803571 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-aa665cd8-26e8-4a5e-b3b7-33406e2a9eb8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8a1c79ab449742c3b1aaa6639b7ceaf5'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:22.816586 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-aa665cd8-26e8-4a5e-b3b7-33406e2a9eb8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8a1c79ab449742c3b1aaa6639b7ceaf5'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:22.817979 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-aa665cd8-26e8-4a5e-b3b7-33406e2a9eb8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8a1c79ab449742c3b1aaa6639b7ceaf5'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:22.818848 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-aa665cd8-26e8-4a5e-b3b7-33406e2a9eb8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8a1c79ab449742c3b1aaa6639b7ceaf5'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:22.836430 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-aa665cd8-26e8-4a5e-b3b7-33406e2a9eb8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8a1c79ab449742c3b1aaa6639b7ceaf5'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:22.838769 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:54:22.885228 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1427/2841] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:54:22 2022] DELETE /identity/v3/users/8a1c79ab449742c3b1aaa6639b7ceaf5 => generated 0 bytes in 296 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:54:22.890817 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823a5264-71e9-4d91-93d2-d89da6bc19a5 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:22.891443 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823a5264-71e9-4d91-93d2-d89da6bc19a5 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:22.891941 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823a5264-71e9-4d91-93d2-d89da6bc19a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:22.930112 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-823a5264-71e9-4d91-93d2-d89da6bc19a5 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:54:22.932150 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-823a5264-71e9-4d91-93d2-d89da6bc19a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:22.996086 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:54:23.036030 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3116706-a181-494d-9407-4bfa9817817b None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:23.036616 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3116706-a181-494d-9407-4bfa9817817b None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:23.036928 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3116706-a181-494d-9407-4bfa9817817b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:23.051000 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1415/2842] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:54:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:54:23.063256 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823a5264-71e9-4d91-93d2-d89da6bc19a5 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:23.066250 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-823a5264-71e9-4d91-93d2-d89da6bc19a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:23.070166 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-e3116706-a181-494d-9407-4bfa9817817b None None] MFA Rules not processed for user `97c8877c74194a6bbbf25b3c8f5b7dbd`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:54:23.072279 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e3116706-a181-494d-9407-4bfa9817817b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:23.154437 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823a5264-71e9-4d91-93d2-d89da6bc19a5 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:23.160421 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-374f2218-8156-4a6f-b811-81ff550c2947 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:23.160853 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374f2218-8156-4a6f-b811-81ff550c2947 service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:23.161359 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374f2218-8156-4a6f-b811-81ff550c2947 service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:23.162043 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374f2218-8156-4a6f-b811-81ff550c2947 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:23.176614 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-374f2218-8156-4a6f-b811-81ff550c2947 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:23.179255 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-374f2218-8156-4a6f-b811-81ff550c2947 service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:23.182093 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1428/2843] 10.222.0.25 () {58 vars in 967 bytes} [Fri Oct 28 09:54:23 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:54:23.196404 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1416/2844] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:54:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:54:23.202993 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3116706-a181-494d-9407-4bfa9817817b None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:23.205567 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e3116706-a181-494d-9407-4bfa9817817b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:23.336974 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3116706-a181-494d-9407-4bfa9817817b None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:23.347699 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6156c1-4e46-4526-9cae-89270eef2778 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c6cffe0d56e45c4b8fb54fbe5df6094', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:23.348783 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6156c1-4e46-4526-9cae-89270eef2778 service swift] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:23.349404 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6156c1-4e46-4526-9cae-89270eef2778 service swift] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:23.350318 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6156c1-4e46-4526-9cae-89270eef2778 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:23.365524 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e6156c1-4e46-4526-9cae-89270eef2778 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:23.367596 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e6156c1-4e46-4526-9cae-89270eef2778 service swift] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:23.380362 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1429/2845] 10.222.0.25 () {60 vars in 1384 bytes} [Fri Oct 28 09:54:23 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 179 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 28 09:54:23.740534 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d436df5-97db-4bec-a142-1907a353bc93 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:23.741396 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d436df5-97db-4bec-a142-1907a353bc93 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:23.741814 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d436df5-97db-4bec-a142-1907a353bc93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:23.764305 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-9d436df5-97db-4bec-a142-1907a353bc93 None None] MFA Rules not processed for user `97c8877c74194a6bbbf25b3c8f5b7dbd`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:54:23.765643 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-9d436df5-97db-4bec-a142-1907a353bc93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:23.774290 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1417/2846] 10.222.0.25 () {58 vars in 1009 bytes} [Fri Oct 28 09:54:23 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:54:23.779608 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6156c1-4e46-4526-9cae-89270eef2778 service swift] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:23.784315 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-8e6156c1-4e46-4526-9cae-89270eef2778 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:23.792387 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d436df5-97db-4bec-a142-1907a353bc93 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:23.802658 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6156c1-4e46-4526-9cae-89270eef2778 service swift] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:23.812300 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d436df5-97db-4bec-a142-1907a353bc93 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:23.812638 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-920094d1-f8d1-43ba-a366-02d35c017d44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:23.813099 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920094d1-f8d1-43ba-a366-02d35c017d44 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:23.813390 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920094d1-f8d1-43ba-a366-02d35c017d44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:23.813709 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920094d1-f8d1-43ba-a366-02d35c017d44 admin admin] PATH_INFO: `/v3/projects/a68cbdfd379248ed9145fdd7a70a17a7` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:23.815558 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-920094d1-f8d1-43ba-a366-02d35c017d44 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a68cbdfd379248ed9145fdd7a70a17a7)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:23.817570 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-920094d1-f8d1-43ba-a366-02d35c017d44 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:23.819844 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4f6f45-b36b-4384-8d07-c8e4c1be1d76 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c6cffe0d56e45c4b8fb54fbe5df6094', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:23.820836 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4f6f45-b36b-4384-8d07-c8e4c1be1d76 service swift] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:23.821310 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4f6f45-b36b-4384-8d07-c8e4c1be1d76 service swift] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:23.821790 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4f6f45-b36b-4384-8d07-c8e4c1be1d76 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:23.839830 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe4f6f45-b36b-4384-8d07-c8e4c1be1d76 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:23.842226 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe4f6f45-b36b-4384-8d07-c8e4c1be1d76 service swift] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:23.856790 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1418/2847] 10.222.0.25 () {60 vars in 1384 bytes} [Fri Oct 28 09:54:23 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 65 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 28 09:54:23.887168 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-920094d1-f8d1-43ba-a366-02d35c017d44 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a68cbdfd379248ed9145fdd7a70a17a7', 'request_id': 'req-920094d1-f8d1-43ba-a366-02d35c017d44'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:23.899912 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-920094d1-f8d1-43ba-a366-02d35c017d44 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a68cbdfd379248ed9145fdd7a70a17a7', 'request_id': 'req-920094d1-f8d1-43ba-a366-02d35c017d44'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:23.912343 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-920094d1-f8d1-43ba-a366-02d35c017d44 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a68cbdfd379248ed9145fdd7a70a17a7', 'request_id': 'req-920094d1-f8d1-43ba-a366-02d35c017d44'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:23.924875 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:54:23.933486 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-920094d1-f8d1-43ba-a366-02d35c017d44 admin admin] The token cache is being invalidate because project a68cbdfd379248ed9145fdd7a70a17a7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:54:23.934195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-920094d1-f8d1-43ba-a366-02d35c017d44 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:23.935195 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-920094d1-f8d1-43ba-a366-02d35c017d44 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:23.937555 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1430/2848] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:54:23 2022] DELETE /identity/v3/projects/a68cbdfd379248ed9145fdd7a70a17a7 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:54:23.942450 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4f6f45-b36b-4384-8d07-c8e4c1be1d76 service swift] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:23.954446 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4f6f45-b36b-4384-8d07-c8e4c1be1d76 service swift] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:24.029598 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2390326-7e2a-4e8d-9fd6-3428bba68668 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:24.030477 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2390326-7e2a-4e8d-9fd6-3428bba68668 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:24.030661 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2390326-7e2a-4e8d-9fd6-3428bba68668 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:24.030905 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2390326-7e2a-4e8d-9fd6-3428bba68668 admin admin] PATH_INFO: `/v3/users/559463366dae442cb501816ceb58b84d` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:24.032432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2390326-7e2a-4e8d-9fd6-3428bba68668 admin admin] RBAC: Authorizing `identity:delete_user(user_id=559463366dae442cb501816ceb58b84d)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:24.034695 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2390326-7e2a-4e8d-9fd6-3428bba68668 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:24.118811 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-c2390326-7e2a-4e8d-9fd6-3428bba68668 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '559463366dae442cb501816ceb58b84d'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:24.129532 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-c2390326-7e2a-4e8d-9fd6-3428bba68668 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '559463366dae442cb501816ceb58b84d'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:24.130365 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-c2390326-7e2a-4e8d-9fd6-3428bba68668 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '559463366dae442cb501816ceb58b84d'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:24.131483 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-c2390326-7e2a-4e8d-9fd6-3428bba68668 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '559463366dae442cb501816ceb58b84d'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:24.133438 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:54:24.146575 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-c2390326-7e2a-4e8d-9fd6-3428bba68668 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '559463366dae442cb501816ceb58b84d'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:24.187119 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1419/2849] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:54:23 2022] DELETE /identity/v3/users/559463366dae442cb501816ceb58b84d => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:54:24.377286 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b820af7-17ea-46bf-aad6-c913e4f3985f None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:24.378626 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b820af7-17ea-46bf-aad6-c913e4f3985f None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:24.378626 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b820af7-17ea-46bf-aad6-c913e4f3985f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:24.408891 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-4b820af7-17ea-46bf-aad6-c913e4f3985f None None] MFA Rules not processed for user `97c8877c74194a6bbbf25b3c8f5b7dbd`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:54:24.411891 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4b820af7-17ea-46bf-aad6-c913e4f3985f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:24.509065 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1431/2850] 10.222.0.25 () {58 vars in 967 bytes} [Fri Oct 28 09:54:24 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:54:24.524257 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2390326-7e2a-4e8d-9fd6-3428bba68668 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:24.526312 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-c2390326-7e2a-4e8d-9fd6-3428bba68668 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:24.636580 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b820af7-17ea-46bf-aad6-c913e4f3985f None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:24.640361 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-4b820af7-17ea-46bf-aad6-c913e4f3985f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:24.661803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2390326-7e2a-4e8d-9fd6-3428bba68668 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:24.670565 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e2cd0a-3620-474d-95ab-32dc1c438ebc service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c6cffe0d56e45c4b8fb54fbe5df6094', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:24.671955 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e2cd0a-3620-474d-95ab-32dc1c438ebc service swift] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:24.672411 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e2cd0a-3620-474d-95ab-32dc1c438ebc service swift] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:24.673234 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e2cd0a-3620-474d-95ab-32dc1c438ebc service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:24.692206 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10e2cd0a-3620-474d-95ab-32dc1c438ebc service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:24.695236 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10e2cd0a-3620-474d-95ab-32dc1c438ebc service swift] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:24.709884 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1420/2851] 10.222.0.25 () {60 vars in 1384 bytes} [Fri Oct 28 09:54:24 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 186 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 28 09:54:24.764443 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b820af7-17ea-46bf-aad6-c913e4f3985f None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:24.771880 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f157bc-f139-4bc5-81b2-aa3018cda377 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:24.772797 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f157bc-f139-4bc5-81b2-aa3018cda377 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:24.773252 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f157bc-f139-4bc5-81b2-aa3018cda377 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:24.773766 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f157bc-f139-4bc5-81b2-aa3018cda377 admin admin] PATH_INFO: `/v3/projects/f64b7401eac942ecb30c28c81076ff60` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:24.775992 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f157bc-f139-4bc5-81b2-aa3018cda377 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f64b7401eac942ecb30c28c81076ff60)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:24.780391 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f157bc-f139-4bc5-81b2-aa3018cda377 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:24.865534 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a8f157bc-f139-4bc5-81b2-aa3018cda377 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f64b7401eac942ecb30c28c81076ff60', 'request_id': 'req-a8f157bc-f139-4bc5-81b2-aa3018cda377'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:24.874172 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a8f157bc-f139-4bc5-81b2-aa3018cda377 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f64b7401eac942ecb30c28c81076ff60', 'request_id': 'req-a8f157bc-f139-4bc5-81b2-aa3018cda377'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:24.889012 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a8f157bc-f139-4bc5-81b2-aa3018cda377 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f64b7401eac942ecb30c28c81076ff60', 'request_id': 'req-a8f157bc-f139-4bc5-81b2-aa3018cda377'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:24.909514 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a8f157bc-f139-4bc5-81b2-aa3018cda377 admin admin] The token cache is being invalidate because project f64b7401eac942ecb30c28c81076ff60 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:54:24.909885 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a8f157bc-f139-4bc5-81b2-aa3018cda377 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:24.910551 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-a8f157bc-f139-4bc5-81b2-aa3018cda377 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:24.916895 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1432/2852] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:54:24 2022] DELETE /identity/v3/projects/f64b7401eac942ecb30c28c81076ff60 => generated 0 bytes in 278 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:54:24.919397 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e932fbc-0413-4863-9c17-dbfa73e20795 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:24.919999 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e932fbc-0413-4863-9c17-dbfa73e20795 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:24.920432 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e932fbc-0413-4863-9c17-dbfa73e20795 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:24.947493 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-2e932fbc-0413-4863-9c17-dbfa73e20795 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:54:24.950654 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-2e932fbc-0413-4863-9c17-dbfa73e20795 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:25.050061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1421/2853] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:54:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:54:25.060051 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d64173-6a8e-4661-ada2-e7576b78fdb0 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:25.060833 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d64173-6a8e-4661-ada2-e7576b78fdb0 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:25.061344 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d64173-6a8e-4661-ada2-e7576b78fdb0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:25.100841 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-e0d64173-6a8e-4661-ada2-e7576b78fdb0 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:54:25.111996 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e0d64173-6a8e-4661-ada2-e7576b78fdb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:25.130509 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1433/2854] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:54:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 73 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:54:25.138310 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e932fbc-0413-4863-9c17-dbfa73e20795 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:25.156562 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e932fbc-0413-4863-9c17-dbfa73e20795 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:25.163988 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbfbb4ea-ab98-4dbd-b931-80c24ded8e83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:25.165806 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbfbb4ea-ab98-4dbd-b931-80c24ded8e83 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:25.166511 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbfbb4ea-ab98-4dbd-b931-80c24ded8e83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:25.167470 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbfbb4ea-ab98-4dbd-b931-80c24ded8e83 admin admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:25.169345 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbfbb4ea-ab98-4dbd-b931-80c24ded8e83 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:54:25.169888 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbfbb4ea-ab98-4dbd-b931-80c24ded8e83 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:25.172466 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbfbb4ea-ab98-4dbd-b931-80c24ded8e83 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:25.195665 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1422/2855] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:54:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:54:25.201574 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d64173-6a8e-4661-ada2-e7576b78fdb0 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:25.214670 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d64173-6a8e-4661-ada2-e7576b78fdb0 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:25.222625 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2520b8f9-08b2-4362-95f9-072cb618fa83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:25.223723 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2520b8f9-08b2-4362-95f9-072cb618fa83 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:25.224174 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2520b8f9-08b2-4362-95f9-072cb618fa83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:25.224704 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2520b8f9-08b2-4362-95f9-072cb618fa83 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:25.225470 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2520b8f9-08b2-4362-95f9-072cb618fa83 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:25.227936 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2520b8f9-08b2-4362-95f9-072cb618fa83 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:25.285800 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1434/2856] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:54:25 2022] POST /identity/v3/projects => generated 373 bytes in 85 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:54:25.295702 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbfbb4ea-ab98-4dbd-b931-80c24ded8e83 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:25.307667 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbfbb4ea-ab98-4dbd-b931-80c24ded8e83 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:25.379237 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:54:25.399655 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-180b0ea1-1239-4bbc-b9f1-699cad2fa2de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:25.400631 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180b0ea1-1239-4bbc-b9f1-699cad2fa2de admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:25.401247 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180b0ea1-1239-4bbc-b9f1-699cad2fa2de admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:25.401247 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180b0ea1-1239-4bbc-b9f1-699cad2fa2de admin admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:25.402065 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-180b0ea1-1239-4bbc-b9f1-699cad2fa2de admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:25.405359 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-180b0ea1-1239-4bbc-b9f1-699cad2fa2de admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:25.442508 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1423/2857] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:54:25 2022] POST /identity/v3/users => generated 287 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Oct 28 09:54:25.446818 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2520b8f9-08b2-4362-95f9-072cb618fa83 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:25.453042 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:54:25.457393 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2520b8f9-08b2-4362-95f9-072cb618fa83 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:25.464849 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1bd2be9-5adb-4388-ba04-374737f79c84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:25.465582 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1bd2be9-5adb-4388-ba04-374737f79c84 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:25.467229 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1bd2be9-5adb-4388-ba04-374737f79c84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:25.467742 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1bd2be9-5adb-4388-ba04-374737f79c84 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:25.468539 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1bd2be9-5adb-4388-ba04-374737f79c84 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:25.470457 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1bd2be9-5adb-4388-ba04-374737f79c84 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:25.492985 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1435/2858] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:54:25 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:54:25.495122 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-180b0ea1-1239-4bbc-b9f1-699cad2fa2de admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:25.507882 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-180b0ea1-1239-4bbc-b9f1-699cad2fa2de admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:25.515357 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2098a530-adea-4c97-bfcd-551b6de4d744 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:25.516547 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2098a530-adea-4c97-bfcd-551b6de4d744 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:25.516721 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2098a530-adea-4c97-bfcd-551b6de4d744 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:25.516984 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2098a530-adea-4c97-bfcd-551b6de4d744 admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:25.517923 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2098a530-adea-4c97-bfcd-551b6de4d744 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:25.520671 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2098a530-adea-4c97-bfcd-551b6de4d744 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:25.538516 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1424/2859] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:54:25 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:54:25.542236 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1bd2be9-5adb-4388-ba04-374737f79c84 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:25.560702 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1bd2be9-5adb-4388-ba04-374737f79c84 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:25.567200 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d684f44c-b2bd-4ffe-a8ac-9125410fa129 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:25.567805 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d684f44c-b2bd-4ffe-a8ac-9125410fa129 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:25.568104 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d684f44c-b2bd-4ffe-a8ac-9125410fa129 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:25.568526 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d684f44c-b2bd-4ffe-a8ac-9125410fa129 admin admin] PATH_INFO: `/v3/projects/c9387871bcfd499496f67871a14caf73/users/aec5f8c400f04bd8aac834c2a6b139ed/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:25.592979 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d684f44c-b2bd-4ffe-a8ac-9125410fa129 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c9387871bcfd499496f67871a14caf73, user_id=aec5f8c400f04bd8aac834c2a6b139ed, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:25.597215 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d684f44c-b2bd-4ffe-a8ac-9125410fa129 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:25.622758 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1436/2860] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:54:25 2022] PUT /identity/v3/projects/c9387871bcfd499496f67871a14caf73/users/aec5f8c400f04bd8aac834c2a6b139ed/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 79 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:54:25.638072 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4513086-f43d-467a-94a8-f36a8db04586 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:25.639004 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4513086-f43d-467a-94a8-f36a8db04586 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:25.639260 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4513086-f43d-467a-94a8-f36a8db04586 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:25.676220 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-d4513086-f43d-467a-94a8-f36a8db04586 None None] MFA Rules not processed for user `aec5f8c400f04bd8aac834c2a6b139ed`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:54:25.677428 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-d4513086-f43d-467a-94a8-f36a8db04586 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:25.760715 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1425/2861] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:54:25 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:54:25.786156 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d684f44c-b2bd-4ffe-a8ac-9125410fa129 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:25.789175 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-d684f44c-b2bd-4ffe-a8ac-9125410fa129 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:25.865168 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d684f44c-b2bd-4ffe-a8ac-9125410fa129 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:25.869657 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a84bbea-0829-4495-8599-6edd36deae05 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:25.870304 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a84bbea-0829-4495-8599-6edd36deae05 service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:25.870717 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a84bbea-0829-4495-8599-6edd36deae05 service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:25.871002 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a84bbea-0829-4495-8599-6edd36deae05 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:25.888328 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a84bbea-0829-4495-8599-6edd36deae05 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:25.888328 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a84bbea-0829-4495-8599-6edd36deae05 service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:25.898785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1437/2862] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:54:25 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:54:25.943964 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4513086-f43d-467a-94a8-f36a8db04586 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:26.036688 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4513086-f43d-467a-94a8-f36a8db04586 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:26.046663 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1759c238-1d5c-4926-8a8e-8b15a06ec923 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:26.047681 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1759c238-1d5c-4926-8a8e-8b15a06ec923 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:26.048213 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1759c238-1d5c-4926-8a8e-8b15a06ec923 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:26.049374 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1759c238-1d5c-4926-8a8e-8b15a06ec923 admin admin] PATH_INFO: `/v3/users/aec5f8c400f04bd8aac834c2a6b139ed` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:26.050988 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1759c238-1d5c-4926-8a8e-8b15a06ec923 admin admin] RBAC: Authorizing `identity:delete_user(user_id=aec5f8c400f04bd8aac834c2a6b139ed)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:26.053132 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1759c238-1d5c-4926-8a8e-8b15a06ec923 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:26.124112 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-1759c238-1d5c-4926-8a8e-8b15a06ec923 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'aec5f8c400f04bd8aac834c2a6b139ed'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:26.134519 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-1759c238-1d5c-4926-8a8e-8b15a06ec923 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'aec5f8c400f04bd8aac834c2a6b139ed'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:26.135631 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-1759c238-1d5c-4926-8a8e-8b15a06ec923 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'aec5f8c400f04bd8aac834c2a6b139ed'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:26.139674 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-1759c238-1d5c-4926-8a8e-8b15a06ec923 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'aec5f8c400f04bd8aac834c2a6b139ed'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:26.147128 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=88,keystone:INSERT=4,keystone:DELETE=8 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:54:26.152372 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-1759c238-1d5c-4926-8a8e-8b15a06ec923 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'aec5f8c400f04bd8aac834c2a6b139ed'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:26.188038 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1426/2863] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:54:25 2022] DELETE /identity/v3/users/aec5f8c400f04bd8aac834c2a6b139ed => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:54:26.193270 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa0b5b7-bc03-4210-965d-aa1878077f74 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:26.193618 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa0b5b7-bc03-4210-965d-aa1878077f74 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:26.193902 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa0b5b7-bc03-4210-965d-aa1878077f74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:26.219533 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-5aa0b5b7-bc03-4210-965d-aa1878077f74 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:54:26.220885 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-5aa0b5b7-bc03-4210-965d-aa1878077f74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:26.319050 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1438/2864] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:54:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:54:26.333757 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1759c238-1d5c-4926-8a8e-8b15a06ec923 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:26.337762 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-1759c238-1d5c-4926-8a8e-8b15a06ec923 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:26.438156 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1759c238-1d5c-4926-8a8e-8b15a06ec923 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:26.443382 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c606016-deb8-47e0-a3ba-3bbdaf9d50a6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:26.444054 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c606016-deb8-47e0-a3ba-3bbdaf9d50a6 service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:26.444507 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c606016-deb8-47e0-a3ba-3bbdaf9d50a6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:26.444796 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c606016-deb8-47e0-a3ba-3bbdaf9d50a6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:26.456161 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c606016-deb8-47e0-a3ba-3bbdaf9d50a6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:26.457848 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c606016-deb8-47e0-a3ba-3bbdaf9d50a6 service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:26.470205 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1427/2865] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:54:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:54:26.830344 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa0b5b7-bc03-4210-965d-aa1878077f74 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:26.832386 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-5aa0b5b7-bc03-4210-965d-aa1878077f74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:26.849547 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa0b5b7-bc03-4210-965d-aa1878077f74 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:26.854419 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7662af78-aa7e-429a-80b5-36044c255b15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:26.855417 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7662af78-aa7e-429a-80b5-36044c255b15 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:26.856206 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7662af78-aa7e-429a-80b5-36044c255b15 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:26.857164 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7662af78-aa7e-429a-80b5-36044c255b15 admin admin] PATH_INFO: `/v3/projects/c9387871bcfd499496f67871a14caf73` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:26.858782 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7662af78-aa7e-429a-80b5-36044c255b15 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c9387871bcfd499496f67871a14caf73)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:26.860705 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7662af78-aa7e-429a-80b5-36044c255b15 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:26.917811 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7662af78-aa7e-429a-80b5-36044c255b15 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c9387871bcfd499496f67871a14caf73', 'request_id': 'req-7662af78-aa7e-429a-80b5-36044c255b15'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:26.926660 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7662af78-aa7e-429a-80b5-36044c255b15 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c9387871bcfd499496f67871a14caf73', 'request_id': 'req-7662af78-aa7e-429a-80b5-36044c255b15'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:26.936628 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7662af78-aa7e-429a-80b5-36044c255b15 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c9387871bcfd499496f67871a14caf73', 'request_id': 'req-7662af78-aa7e-429a-80b5-36044c255b15'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:26.942336 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:54:26.951862 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7662af78-aa7e-429a-80b5-36044c255b15 admin admin] The token cache is being invalidate because project c9387871bcfd499496f67871a14caf73 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:54:26.952233 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7662af78-aa7e-429a-80b5-36044c255b15 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:26.953036 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-7662af78-aa7e-429a-80b5-36044c255b15 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:26.955009 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1439/2866] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:54:26 2022] DELETE /identity/v3/projects/c9387871bcfd499496f67871a14caf73 => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:54:26.960538 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2c4328-8a23-4908-af79-438bcf8d3bfa None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:26.960855 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2c4328-8a23-4908-af79-438bcf8d3bfa None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:26.961041 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2c4328-8a23-4908-af79-438bcf8d3bfa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:26.983281 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-fc2c4328-8a23-4908-af79-438bcf8d3bfa None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:54:26.984550 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-fc2c4328-8a23-4908-af79-438bcf8d3bfa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:27.053419 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1428/2867] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:54:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:54:27.062767 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859728ba-9efa-4885-ae19-6fc61f99572a None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:27.063136 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859728ba-9efa-4885-ae19-6fc61f99572a None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:27.063422 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859728ba-9efa-4885-ae19-6fc61f99572a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:27.084441 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-859728ba-9efa-4885-ae19-6fc61f99572a None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:54:27.085576 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-859728ba-9efa-4885-ae19-6fc61f99572a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:27.094696 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1440/2868] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:54:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:54:27.099363 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc2c4328-8a23-4908-af79-438bcf8d3bfa None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:27.114359 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc2c4328-8a23-4908-af79-438bcf8d3bfa None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:27.120077 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13086a09-9161-4056-956d-47b840a4704a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:27.120803 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13086a09-9161-4056-956d-47b840a4704a admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:27.121166 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13086a09-9161-4056-956d-47b840a4704a admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:27.121542 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13086a09-9161-4056-956d-47b840a4704a admin admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:27.122340 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13086a09-9161-4056-956d-47b840a4704a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:54:27.122758 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13086a09-9161-4056-956d-47b840a4704a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:27.124685 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13086a09-9161-4056-956d-47b840a4704a admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:27.138942 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1429/2869] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:54:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:54:27.143453 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-859728ba-9efa-4885-ae19-6fc61f99572a None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:27.153668 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-859728ba-9efa-4885-ae19-6fc61f99572a None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:27.158740 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0432fae-5e3a-45a2-b4cc-4cb94c4fc1ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:27.159364 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0432fae-5e3a-45a2-b4cc-4cb94c4fc1ec admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:27.159646 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0432fae-5e3a-45a2-b4cc-4cb94c4fc1ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:27.159921 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0432fae-5e3a-45a2-b4cc-4cb94c4fc1ec admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:27.160567 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0432fae-5e3a-45a2-b4cc-4cb94c4fc1ec admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:27.162530 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0432fae-5e3a-45a2-b4cc-4cb94c4fc1ec admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:27.194516 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1441/2870] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:54:27 2022] POST /identity/v3/projects => generated 391 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:54:27.201562 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13086a09-9161-4056-956d-47b840a4704a admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:27.219015 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13086a09-9161-4056-956d-47b840a4704a admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:27.294111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2664c939-23c4-4145-8dd5-3aa0ad05868c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:27.294765 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2664c939-23c4-4145-8dd5-3aa0ad05868c admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:27.295084 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2664c939-23c4-4145-8dd5-3aa0ad05868c admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:27.295365 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2664c939-23c4-4145-8dd5-3aa0ad05868c admin admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:27.296025 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2664c939-23c4-4145-8dd5-3aa0ad05868c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:27.298258 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2664c939-23c4-4145-8dd5-3aa0ad05868c admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:27.311112 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:54:27.328528 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1430/2871] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:54:27 2022] POST /identity/v3/users => generated 296 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:54:27.334794 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0432fae-5e3a-45a2-b4cc-4cb94c4fc1ec admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:27.346382 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0432fae-5e3a-45a2-b4cc-4cb94c4fc1ec admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:27.352459 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a60775e-40e4-4d1c-9fc0-65f1e9aca66f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:27.353427 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a60775e-40e4-4d1c-9fc0-65f1e9aca66f admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:27.353892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a60775e-40e4-4d1c-9fc0-65f1e9aca66f admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:27.354321 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a60775e-40e4-4d1c-9fc0-65f1e9aca66f admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:27.355250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a60775e-40e4-4d1c-9fc0-65f1e9aca66f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:27.357295 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a60775e-40e4-4d1c-9fc0-65f1e9aca66f admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:27.370666 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1442/2872] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:54:27 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:54:27.373277 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2664c939-23c4-4145-8dd5-3aa0ad05868c admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:27.375438 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-2664c939-23c4-4145-8dd5-3aa0ad05868c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:27.377122 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a60775e-40e4-4d1c-9fc0-65f1e9aca66f admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:27.389237 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2664c939-23c4-4145-8dd5-3aa0ad05868c admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:27.390179 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a60775e-40e4-4d1c-9fc0-65f1e9aca66f admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:27.395168 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53c17490-1091-48df-b0d7-5ff06a891daa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:27.395610 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f399320-1bc1-41bb-89c6-8c1997f2c0b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:27.396055 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c17490-1091-48df-b0d7-5ff06a891daa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:27.396244 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f399320-1bc1-41bb-89c6-8c1997f2c0b6 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:27.396500 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f399320-1bc1-41bb-89c6-8c1997f2c0b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:27.396678 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c17490-1091-48df-b0d7-5ff06a891daa admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:27.396864 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f399320-1bc1-41bb-89c6-8c1997f2c0b6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:27.397039 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c17490-1091-48df-b0d7-5ff06a891daa admin admin] PATH_INFO: `/v3/users/4d088636881e45559699462268bd0411` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:27.397420 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f399320-1bc1-41bb-89c6-8c1997f2c0b6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:27.398522 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53c17490-1091-48df-b0d7-5ff06a891daa admin admin] RBAC: Authorizing `identity:delete_user(user_id=4d088636881e45559699462268bd0411)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:27.399103 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f399320-1bc1-41bb-89c6-8c1997f2c0b6 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:27.400822 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53c17490-1091-48df-b0d7-5ff06a891daa admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:27.410967 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1443/2873] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:54:27 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Oct 28 09:54:27.416284 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f399320-1bc1-41bb-89c6-8c1997f2c0b6 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:27.436607 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f399320-1bc1-41bb-89c6-8c1997f2c0b6 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:27.445760 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad434e4e-b8d3-44e0-862c-fd434e4df10b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:27.447120 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad434e4e-b8d3-44e0-862c-fd434e4df10b admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:27.447577 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad434e4e-b8d3-44e0-862c-fd434e4df10b admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:27.448037 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad434e4e-b8d3-44e0-862c-fd434e4df10b admin admin] PATH_INFO: `/v3/projects/2270d3b706994f90abcf576afacbd1c6/users/ce2fe7103bdb45db982583a1f5239aef/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:27.471923 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-53c17490-1091-48df-b0d7-5ff06a891daa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4d088636881e45559699462268bd0411'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:27.479645 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad434e4e-b8d3-44e0-862c-fd434e4df10b admin admin] RBAC: Authorizing `identity:create_grant(project_id=2270d3b706994f90abcf576afacbd1c6, user_id=ce2fe7103bdb45db982583a1f5239aef, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:27.482645 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-53c17490-1091-48df-b0d7-5ff06a891daa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4d088636881e45559699462268bd0411'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:27.483327 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad434e4e-b8d3-44e0-862c-fd434e4df10b admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:27.483558 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-53c17490-1091-48df-b0d7-5ff06a891daa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4d088636881e45559699462268bd0411'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:27.484160 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-53c17490-1091-48df-b0d7-5ff06a891daa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4d088636881e45559699462268bd0411'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:27.494367 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-53c17490-1091-48df-b0d7-5ff06a891daa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4d088636881e45559699462268bd0411'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:27.507045 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1444/2874] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:54:27 2022] PUT /identity/v3/projects/2270d3b706994f90abcf576afacbd1c6/users/ce2fe7103bdb45db982583a1f5239aef/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 92 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:54:27.516454 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1431/2875] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:54:27 2022] DELETE /identity/v3/users/4d088636881e45559699462268bd0411 => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:54:27.521928 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de79c5aa-8099-46a4-a4c7-2c51c80bde77 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:27.522434 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de79c5aa-8099-46a4-a4c7-2c51c80bde77 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:27.522828 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de79c5aa-8099-46a4-a4c7-2c51c80bde77 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:27.526293 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d988ed10-95b2-4baf-a04c-c889bda19c58 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:27.526619 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d988ed10-95b2-4baf-a04c-c889bda19c58 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:27.526841 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d988ed10-95b2-4baf-a04c-c889bda19c58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:27.551446 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-de79c5aa-8099-46a4-a4c7-2c51c80bde77 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:54:27.552998 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-de79c5aa-8099-46a4-a4c7-2c51c80bde77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:27.582171 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-d988ed10-95b2-4baf-a04c-c889bda19c58 None None] MFA Rules not processed for user `ce2fe7103bdb45db982583a1f5239aef`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:54:27.583581 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-d988ed10-95b2-4baf-a04c-c889bda19c58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:27.641372 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1445/2876] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:54:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:54:27.656355 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de79c5aa-8099-46a4-a4c7-2c51c80bde77 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:27.658178 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-de79c5aa-8099-46a4-a4c7-2c51c80bde77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:27.679948 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1432/2877] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:54:27 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:54:27.693452 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d988ed10-95b2-4baf-a04c-c889bda19c58 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:27.695232 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-d988ed10-95b2-4baf-a04c-c889bda19c58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:27.739738 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de79c5aa-8099-46a4-a4c7-2c51c80bde77 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:27.751772 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7aa7c8c-7ccd-4cb1-94c1-6fbe414a5dd7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:27.752755 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7aa7c8c-7ccd-4cb1-94c1-6fbe414a5dd7 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:27.753251 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7aa7c8c-7ccd-4cb1-94c1-6fbe414a5dd7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:27.753715 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7aa7c8c-7ccd-4cb1-94c1-6fbe414a5dd7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:27.772347 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7aa7c8c-7ccd-4cb1-94c1-6fbe414a5dd7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:27.773991 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7aa7c8c-7ccd-4cb1-94c1-6fbe414a5dd7 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:27.795463 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1446/2878] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:54:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:54:27.799391 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d988ed10-95b2-4baf-a04c-c889bda19c58 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:27.804445 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7be7b74-efa7-4622-bf8b-6056c688ca93 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:27.805375 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7be7b74-efa7-4622-bf8b-6056c688ca93 service cinder] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:27.805841 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7be7b74-efa7-4622-bf8b-6056c688ca93 service cinder] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:27.806341 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7be7b74-efa7-4622-bf8b-6056c688ca93 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:27.819339 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7be7b74-efa7-4622-bf8b-6056c688ca93 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:27.821123 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7be7b74-efa7-4622-bf8b-6056c688ca93 service cinder] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:27.829936 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:54:27.839949 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1433/2879] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:54:27 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:54:28.071591 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7aa7c8c-7ccd-4cb1-94c1-6fbe414a5dd7 service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:28.073962 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-b7aa7c8c-7ccd-4cb1-94c1-6fbe414a5dd7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:28.107153 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7aa7c8c-7ccd-4cb1-94c1-6fbe414a5dd7 service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:28.127492 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-334e1f7e-cfdf-4b40-a26a-db64f52e3c11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:28.128667 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334e1f7e-cfdf-4b40-a26a-db64f52e3c11 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:28.129174 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334e1f7e-cfdf-4b40-a26a-db64f52e3c11 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:28.129685 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334e1f7e-cfdf-4b40-a26a-db64f52e3c11 admin admin] PATH_INFO: `/v3/projects/1d832ea68ebb4c60b962d0cd76f787b8` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:28.132051 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-334e1f7e-cfdf-4b40-a26a-db64f52e3c11 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1d832ea68ebb4c60b962d0cd76f787b8)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:28.136005 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-334e1f7e-cfdf-4b40-a26a-db64f52e3c11 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:28.181540 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:54:28.235210 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-334e1f7e-cfdf-4b40-a26a-db64f52e3c11 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1d832ea68ebb4c60b962d0cd76f787b8', 'request_id': 'req-334e1f7e-cfdf-4b40-a26a-db64f52e3c11'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:28.256578 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-334e1f7e-cfdf-4b40-a26a-db64f52e3c11 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1d832ea68ebb4c60b962d0cd76f787b8', 'request_id': 'req-334e1f7e-cfdf-4b40-a26a-db64f52e3c11'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:28.278666 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-334e1f7e-cfdf-4b40-a26a-db64f52e3c11 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1d832ea68ebb4c60b962d0cd76f787b8', 'request_id': 'req-334e1f7e-cfdf-4b40-a26a-db64f52e3c11'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:28.313727 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-334e1f7e-cfdf-4b40-a26a-db64f52e3c11 admin admin] The token cache is being invalidate because project 1d832ea68ebb4c60b962d0cd76f787b8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:54:28.314622 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-334e1f7e-cfdf-4b40-a26a-db64f52e3c11 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:28.318297 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-334e1f7e-cfdf-4b40-a26a-db64f52e3c11 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:28.322199 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1447/2880] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:54:28 2022] DELETE /identity/v3/projects/1d832ea68ebb4c60b962d0cd76f787b8 => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:54:38.301796 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=16,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:54:42.583001 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b939c2b9-c336-4b67-af2f-6644bc66d929 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:42.583669 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b939c2b9-c336-4b67-af2f-6644bc66d929 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:42.583940 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b939c2b9-c336-4b67-af2f-6644bc66d929 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:42.607310 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-b939c2b9-c336-4b67-af2f-6644bc66d929 None None] MFA Rules not processed for user `97c8877c74194a6bbbf25b3c8f5b7dbd`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:54:42.608771 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-b939c2b9-c336-4b67-af2f-6644bc66d929 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:42.676293 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1434/2881] 10.222.0.25 () {58 vars in 1009 bytes} [Fri Oct 28 09:54:42 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:54:42.690556 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-334e1f7e-cfdf-4b40-a26a-db64f52e3c11 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:42.692101 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-334e1f7e-cfdf-4b40-a26a-db64f52e3c11 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:42.775451 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-334e1f7e-cfdf-4b40-a26a-db64f52e3c11 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:42.781159 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7d88bd-53da-4eb2-a6af-6710a2d1e4ba service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c6cffe0d56e45c4b8fb54fbe5df6094', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:42.781788 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7d88bd-53da-4eb2-a6af-6710a2d1e4ba service swift] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:42.782071 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7d88bd-53da-4eb2-a6af-6710a2d1e4ba service swift] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:42.782345 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7d88bd-53da-4eb2-a6af-6710a2d1e4ba service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:42.791585 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7d88bd-53da-4eb2-a6af-6710a2d1e4ba service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:42.793747 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7d88bd-53da-4eb2-a6af-6710a2d1e4ba service swift] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:42.803967 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1448/2882] 10.222.0.25 () {60 vars in 1384 bytes} [Fri Oct 28 09:54:42 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Oct 28 09:54:49.501934 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b939c2b9-c336-4b67-af2f-6644bc66d929 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:49.508851 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-b939c2b9-c336-4b67-af2f-6644bc66d929 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:49.679607 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b939c2b9-c336-4b67-af2f-6644bc66d929 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:49.687260 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2757fb-d80f-4c23-a23c-c63987de9953 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:49.687963 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2757fb-d80f-4c23-a23c-c63987de9953 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:49.688410 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2757fb-d80f-4c23-a23c-c63987de9953 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:49.688748 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2757fb-d80f-4c23-a23c-c63987de9953 admin admin] PATH_INFO: `/v3/users/ce2fe7103bdb45db982583a1f5239aef` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:49.690278 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b2757fb-d80f-4c23-a23c-c63987de9953 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ce2fe7103bdb45db982583a1f5239aef)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:49.692675 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b2757fb-d80f-4c23-a23c-c63987de9953 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:49.838719 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-4b2757fb-d80f-4c23-a23c-c63987de9953 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ce2fe7103bdb45db982583a1f5239aef'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:49.851235 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-4b2757fb-d80f-4c23-a23c-c63987de9953 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ce2fe7103bdb45db982583a1f5239aef'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:49.852502 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-4b2757fb-d80f-4c23-a23c-c63987de9953 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ce2fe7103bdb45db982583a1f5239aef'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:49.853579 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-4b2757fb-d80f-4c23-a23c-c63987de9953 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ce2fe7103bdb45db982583a1f5239aef'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:49.866863 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-4b2757fb-d80f-4c23-a23c-c63987de9953 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ce2fe7103bdb45db982583a1f5239aef'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:49.896273 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1435/2883] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:54:49 2022] DELETE /identity/v3/users/ce2fe7103bdb45db982583a1f5239aef => generated 0 bytes in 396 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:54:49.901804 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eec39c6-225a-4d3f-a1bb-917c07ead53d None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:49.903027 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eec39c6-225a-4d3f-a1bb-917c07ead53d None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:49.903503 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eec39c6-225a-4d3f-a1bb-917c07ead53d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:49.935100 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-6eec39c6-225a-4d3f-a1bb-917c07ead53d None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:54:49.937078 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-6eec39c6-225a-4d3f-a1bb-917c07ead53d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:50.048425 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1449/2884] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:54:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:54:50.058193 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2757fb-d80f-4c23-a23c-c63987de9953 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:50.060453 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-4b2757fb-d80f-4c23-a23c-c63987de9953 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:50.103082 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:54:50.151323 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2757fb-d80f-4c23-a23c-c63987de9953 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:50.157046 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86ddf7ca-3d6c-48be-9cbe-bfd9594351d4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:50.157702 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ddf7ca-3d6c-48be-9cbe-bfd9594351d4 service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:50.157980 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ddf7ca-3d6c-48be-9cbe-bfd9594351d4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:50.158256 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ddf7ca-3d6c-48be-9cbe-bfd9594351d4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:50.169171 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86ddf7ca-3d6c-48be-9cbe-bfd9594351d4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:50.170704 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86ddf7ca-3d6c-48be-9cbe-bfd9594351d4 service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:50.186160 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1436/2885] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:54:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:54:50.596353 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eec39c6-225a-4d3f-a1bb-917c07ead53d None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:54:50.598123 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-6eec39c6-225a-4d3f-a1bb-917c07ead53d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:54:50.619160 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eec39c6-225a-4d3f-a1bb-917c07ead53d None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:54:50.652529 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c68197-e99d-4024-bb3b-f887b320e8af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:54:50.653655 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c68197-e99d-4024-bb3b-f887b320e8af admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:54:50.654297 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c68197-e99d-4024-bb3b-f887b320e8af admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:54:50.654547 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c68197-e99d-4024-bb3b-f887b320e8af admin admin] PATH_INFO: `/v3/projects/2270d3b706994f90abcf576afacbd1c6` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:54:50.656164 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c68197-e99d-4024-bb3b-f887b320e8af admin admin] RBAC: Authorizing `identity:delete_project(project_id=2270d3b706994f90abcf576afacbd1c6)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:54:50.658945 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c68197-e99d-4024-bb3b-f887b320e8af admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:54:50.738533 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-75c68197-e99d-4024-bb3b-f887b320e8af admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2270d3b706994f90abcf576afacbd1c6', 'request_id': 'req-75c68197-e99d-4024-bb3b-f887b320e8af'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:50.748986 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-75c68197-e99d-4024-bb3b-f887b320e8af admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2270d3b706994f90abcf576afacbd1c6', 'request_id': 'req-75c68197-e99d-4024-bb3b-f887b320e8af'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:50.762586 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-75c68197-e99d-4024-bb3b-f887b320e8af admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2270d3b706994f90abcf576afacbd1c6', 'request_id': 'req-75c68197-e99d-4024-bb3b-f887b320e8af'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:50.780844 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-75c68197-e99d-4024-bb3b-f887b320e8af admin admin] The token cache is being invalidate because project 2270d3b706994f90abcf576afacbd1c6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:54:50.781308 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-75c68197-e99d-4024-bb3b-f887b320e8af admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:50.782297 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-75c68197-e99d-4024-bb3b-f887b320e8af admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:54:50.798652 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1450/2886] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:54:50 2022] DELETE /identity/v3/projects/2270d3b706994f90abcf576afacbd1c6 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:55:00.178130 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=15 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:55:00.775510 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=86,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:55:01.214956 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add9e045-8f43-480f-991e-bfc679b02b16 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:55:01.215568 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add9e045-8f43-480f-991e-bfc679b02b16 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:55:01.215697 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add9e045-8f43-480f-991e-bfc679b02b16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:55:01.238358 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-add9e045-8f43-480f-991e-bfc679b02b16 None None] MFA Rules not processed for user `97c8877c74194a6bbbf25b3c8f5b7dbd`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:55:01.239951 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-add9e045-8f43-480f-991e-bfc679b02b16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:55:01.309527 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1437/2887] 10.222.0.25 () {58 vars in 967 bytes} [Fri Oct 28 09:55:01 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:55:01.326160 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c68197-e99d-4024-bb3b-f887b320e8af admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:55:01.328991 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-75c68197-e99d-4024-bb3b-f887b320e8af admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:55:01.418156 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c68197-e99d-4024-bb3b-f887b320e8af admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:55:01.424845 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6cae3a-4bca-4ab8-937d-c7596b1c2cc2 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9c6cffe0d56e45c4b8fb54fbe5df6094', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:55:01.425697 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6cae3a-4bca-4ab8-937d-c7596b1c2cc2 service swift] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:55:01.426077 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6cae3a-4bca-4ab8-937d-c7596b1c2cc2 service swift] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:55:01.426518 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6cae3a-4bca-4ab8-937d-c7596b1c2cc2 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:55:01.436975 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd6cae3a-4bca-4ab8-937d-c7596b1c2cc2 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:55:01.438810 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd6cae3a-4bca-4ab8-937d-c7596b1c2cc2 service swift] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:55:01.449941 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1451/2888] 10.222.0.25 () {60 vars in 1384 bytes} [Fri Oct 28 09:55:01 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Oct 28 09:55:03.695898 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add9e045-8f43-480f-991e-bfc679b02b16 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:55:03.697597 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-add9e045-8f43-480f-991e-bfc679b02b16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:55:03.777874 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add9e045-8f43-480f-991e-bfc679b02b16 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:55:03.782534 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6bc27d-d84b-46b3-9998-d53f7a4fa839 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:55:03.783196 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6bc27d-d84b-46b3-9998-d53f7a4fa839 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:55:03.783478 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6bc27d-d84b-46b3-9998-d53f7a4fa839 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:55:03.783756 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6bc27d-d84b-46b3-9998-d53f7a4fa839 admin admin] PATH_INFO: `/v3/users/a3c6fa6f402143b598108c6cb2aedf78` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:55:03.784955 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6bc27d-d84b-46b3-9998-d53f7a4fa839 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a3c6fa6f402143b598108c6cb2aedf78)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:55:03.787150 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6bc27d-d84b-46b3-9998-d53f7a4fa839 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:55:03.848882 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-4e6bc27d-d84b-46b3-9998-d53f7a4fa839 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a3c6fa6f402143b598108c6cb2aedf78'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:55:03.857986 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-4e6bc27d-d84b-46b3-9998-d53f7a4fa839 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a3c6fa6f402143b598108c6cb2aedf78'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:55:03.859009 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-4e6bc27d-d84b-46b3-9998-d53f7a4fa839 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a3c6fa6f402143b598108c6cb2aedf78'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:55:03.859640 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-4e6bc27d-d84b-46b3-9998-d53f7a4fa839 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a3c6fa6f402143b598108c6cb2aedf78'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:55:03.868743 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-4e6bc27d-d84b-46b3-9998-d53f7a4fa839 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a3c6fa6f402143b598108c6cb2aedf78'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:55:03.888546 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1438/2889] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:55:03 2022] DELETE /identity/v3/users/a3c6fa6f402143b598108c6cb2aedf78 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:55:04.098712 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6cae3a-4bca-4ab8-937d-c7596b1c2cc2 service swift] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:55:04.119854 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6cae3a-4bca-4ab8-937d-c7596b1c2cc2 service swift] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:55:04.202249 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295026d7-3342-469c-81fa-6e3c44883c65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:55:04.202947 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295026d7-3342-469c-81fa-6e3c44883c65 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:55:04.203234 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295026d7-3342-469c-81fa-6e3c44883c65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:55:04.203513 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295026d7-3342-469c-81fa-6e3c44883c65 admin admin] PATH_INFO: `/v3/projects/6737c612807a468797742417d59da976` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:55:04.204790 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-295026d7-3342-469c-81fa-6e3c44883c65 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6737c612807a468797742417d59da976)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:55:04.206793 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-295026d7-3342-469c-81fa-6e3c44883c65 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:55:04.262509 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-295026d7-3342-469c-81fa-6e3c44883c65 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6737c612807a468797742417d59da976', 'request_id': 'req-295026d7-3342-469c-81fa-6e3c44883c65'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:55:04.269823 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-295026d7-3342-469c-81fa-6e3c44883c65 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6737c612807a468797742417d59da976', 'request_id': 'req-295026d7-3342-469c-81fa-6e3c44883c65'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:55:04.280199 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-295026d7-3342-469c-81fa-6e3c44883c65 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6737c612807a468797742417d59da976', 'request_id': 'req-295026d7-3342-469c-81fa-6e3c44883c65'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:55:04.293378 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-295026d7-3342-469c-81fa-6e3c44883c65 admin admin] The token cache is being invalidate because project 6737c612807a468797742417d59da976 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81577) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:55:04.293679 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-295026d7-3342-469c-81fa-6e3c44883c65 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:55:04.294452 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-295026d7-3342-469c-81fa-6e3c44883c65 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:55:04.296010 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1452/2890] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:55:04 2022] DELETE /identity/v3/projects/6737c612807a468797742417d59da976 => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:55:04.302952 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8b1681-6d8c-463c-be3d-033071f929a4 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:55:04.303433 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8b1681-6d8c-463c-be3d-033071f929a4 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:55:04.303945 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8b1681-6d8c-463c-be3d-033071f929a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:55:04.334660 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-8f8b1681-6d8c-463c-be3d-033071f929a4 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:55:04.336632 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-8f8b1681-6d8c-463c-be3d-033071f929a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:55:04.366610 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:55:04.411177 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1439/2891] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:55:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:55:04.420586 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42d7005-20b3-4421-9711-6688cab18b39 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:55:04.421019 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42d7005-20b3-4421-9711-6688cab18b39 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:55:04.421644 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a42d7005-20b3-4421-9711-6688cab18b39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:55:04.443441 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-a42d7005-20b3-4421-9711-6688cab18b39 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:55:04.444998 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-a42d7005-20b3-4421-9711-6688cab18b39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:55:04.453429 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1453/2892] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:55:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:55:04.457773 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f8b1681-6d8c-463c-be3d-033071f929a4 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:55:04.468016 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f8b1681-6d8c-463c-be3d-033071f929a4 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:55:04.472814 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1231b16c-317c-469b-a909-c529473b4a15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:55:04.473442 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1231b16c-317c-469b-a909-c529473b4a15 admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:55:04.473721 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1231b16c-317c-469b-a909-c529473b4a15 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:55:04.474016 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1231b16c-317c-469b-a909-c529473b4a15 admin admin] PATH_INFO: `/v3/domains` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:55:04.474719 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1231b16c-317c-469b-a909-c529473b4a15 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81576) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Oct 28 09:55:04.475016 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1231b16c-317c-469b-a909-c529473b4a15 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:55:04.476993 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1231b16c-317c-469b-a909-c529473b4a15 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:55:04.490546 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1440/2893] 10.222.0.25 () {58 vars in 1153 bytes} [Fri Oct 28 09:55:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Oct 28 09:55:04.495924 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42d7005-20b3-4421-9711-6688cab18b39 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:55:04.506344 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a42d7005-20b3-4421-9711-6688cab18b39 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:55:04.513321 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69115684-7f26-4570-9024-97fb3bf25c44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:55:04.514078 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69115684-7f26-4570-9024-97fb3bf25c44 admin admin] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:55:04.514574 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69115684-7f26-4570-9024-97fb3bf25c44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:55:04.515030 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69115684-7f26-4570-9024-97fb3bf25c44 admin admin] PATH_INFO: `/v3/projects` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:55:04.516145 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69115684-7f26-4570-9024-97fb3bf25c44 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:55:04.519260 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69115684-7f26-4570-9024-97fb3bf25c44 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:55:04.533300 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:55:04.572424 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1454/2894] 10.222.0.25 () {60 vars in 1153 bytes} [Fri Oct 28 09:55:04 2022] POST /identity/v3/projects => generated 365 bytes in 78 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:55:04.576883 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1231b16c-317c-469b-a909-c529473b4a15 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:55:04.588748 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1231b16c-317c-469b-a909-c529473b4a15 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:55:04.652828 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d61a56-1fd4-4a0a-943b-e4ca24ef3322 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:55:04.653356 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d61a56-1fd4-4a0a-943b-e4ca24ef3322 admin admin] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:55:04.653635 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d61a56-1fd4-4a0a-943b-e4ca24ef3322 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:55:04.653913 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d61a56-1fd4-4a0a-943b-e4ca24ef3322 admin admin] PATH_INFO: `/v3/users` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:55:04.654609 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d61a56-1fd4-4a0a-943b-e4ca24ef3322 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:55:04.656567 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d61a56-1fd4-4a0a-943b-e4ca24ef3322 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:55:04.682061 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1441/2895] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Oct 28 09:55:04 2022] POST /identity/v3/users => generated 283 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Oct 28 09:55:04.686653 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69115684-7f26-4570-9024-97fb3bf25c44 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:55:04.700395 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69115684-7f26-4570-9024-97fb3bf25c44 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:55:04.708552 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b3edf0e-8422-4e63-92e8-4516f0231c14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:55:04.709633 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b3edf0e-8422-4e63-92e8-4516f0231c14 admin admin] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:55:04.710156 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b3edf0e-8422-4e63-92e8-4516f0231c14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:55:04.710713 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b3edf0e-8422-4e63-92e8-4516f0231c14 admin admin] PATH_INFO: `/v3/roles` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:55:04.711923 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b3edf0e-8422-4e63-92e8-4516f0231c14 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:55:04.715019 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b3edf0e-8422-4e63-92e8-4516f0231c14 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:55:04.737488 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1455/2896] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:55:04 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:55:04.743243 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d61a56-1fd4-4a0a-943b-e4ca24ef3322 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:55:04.761915 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d61a56-1fd4-4a0a-943b-e4ca24ef3322 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:55:04.769664 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8696d679-1ba3-481c-8ada-1be6ec4d84bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:55:04.770784 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8696d679-1ba3-481c-8ada-1be6ec4d84bc admin admin] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:55:04.771246 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8696d679-1ba3-481c-8ada-1be6ec4d84bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:55:04.771707 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8696d679-1ba3-481c-8ada-1be6ec4d84bc admin admin] PATH_INFO: `/v3/roles` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:55:04.773057 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8696d679-1ba3-481c-8ada-1be6ec4d84bc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:55:04.776247 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8696d679-1ba3-481c-8ada-1be6ec4d84bc admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:55:04.797539 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1442/2897] 10.222.0.25 () {58 vars in 1122 bytes} [Fri Oct 28 09:55:04 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Oct 28 09:55:04.802862 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b3edf0e-8422-4e63-92e8-4516f0231c14 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:55:04.819992 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b3edf0e-8422-4e63-92e8-4516f0231c14 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:55:04.828397 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee780ad-6c43-4099-b58f-843bf7343b01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:55:04.829509 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee780ad-6c43-4099-b58f-843bf7343b01 admin admin] REQUEST_METHOD: `PUT` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:55:04.830007 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee780ad-6c43-4099-b58f-843bf7343b01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:55:04.830537 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee780ad-6c43-4099-b58f-843bf7343b01 admin admin] PATH_INFO: `/v3/projects/2a1784fc77a543e6adbc6194ceb272d2/users/55c55ff490f04152922385e619f79de9/roles/8943f289c9fe4feba2695b9c90cf3d51` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:55:04.856159 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bee780ad-6c43-4099-b58f-843bf7343b01 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2a1784fc77a543e6adbc6194ceb272d2, user_id=55c55ff490f04152922385e619f79de9, role_id=8943f289c9fe4feba2695b9c90cf3d51)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:55:04.862000 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bee780ad-6c43-4099-b58f-843bf7343b01 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:55:04.884642 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1456/2898] 10.222.0.25 () {60 vars in 1483 bytes} [Fri Oct 28 09:55:04 2022] PUT /identity/v3/projects/2a1784fc77a543e6adbc6194ceb272d2/users/55c55ff490f04152922385e619f79de9/roles/8943f289c9fe4feba2695b9c90cf3d51 => generated 0 bytes in 83 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:55:04.897196 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3d5911-5481-462c-af78-61163b57633d None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:55:04.897501 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3d5911-5481-462c-af78-61163b57633d None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:55:04.897756 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3d5911-5481-462c-af78-61163b57633d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:55:04.930101 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-cb3d5911-5481-462c-af78-61163b57633d None None] MFA Rules not processed for user `55c55ff490f04152922385e619f79de9`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:55:04.931286 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-cb3d5911-5481-462c-af78-61163b57633d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:55:05.019358 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1443/2899] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:55:04 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:55:05.033766 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee780ad-6c43-4099-b58f-843bf7343b01 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:55:05.035766 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-bee780ad-6c43-4099-b58f-843bf7343b01 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:55:05.103154 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee780ad-6c43-4099-b58f-843bf7343b01 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:55:05.107071 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa983f87-022f-4e45-8e3c-854c1b380d8e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1021f1cc73b54d6eaa647a5f49815ef8', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:55:05.107699 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa983f87-022f-4e45-8e3c-854c1b380d8e service cinder] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:55:05.107982 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa983f87-022f-4e45-8e3c-854c1b380d8e service cinder] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:55:05.108263 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa983f87-022f-4e45-8e3c-854c1b380d8e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:55:05.118146 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa983f87-022f-4e45-8e3c-854c1b380d8e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:55:05.119623 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa983f87-022f-4e45-8e3c-854c1b380d8e service cinder] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:55:05.129835 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1457/2900] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:55:05 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:55:09.343906 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3d5911-5481-462c-af78-61163b57633d None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:55:09.345738 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-cb3d5911-5481-462c-af78-61163b57633d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:55:09.375256 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:55:09.420721 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3d5911-5481-462c-af78-61163b57633d None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:55:09.424838 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9fcfa85-f26a-479a-8c03-1e5566fbb833 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:55:09.425472 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fcfa85-f26a-479a-8c03-1e5566fbb833 service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:55:09.425776 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fcfa85-f26a-479a-8c03-1e5566fbb833 service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:55:09.426085 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9fcfa85-f26a-479a-8c03-1e5566fbb833 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:55:09.428056 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-d9fcfa85-f26a-479a-8c03-1e5566fbb833 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:55:09.533193 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9fcfa85-f26a-479a-8c03-1e5566fbb833 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:55:09.534595 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9fcfa85-f26a-479a-8c03-1e5566fbb833 service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:55:09.561735 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1444/2901] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:55:09 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 219 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Oct 28 09:55:15.125178 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=48,keystone:INSERT=2 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:55:19.558360 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=54 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:55:23.832561 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f83f92-7314-4f7b-b248-070bc327cbd1 None None] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:55:23.833500 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f83f92-7314-4f7b-b248-070bc327cbd1 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:55:23.833500 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f83f92-7314-4f7b-b248-070bc327cbd1 None None] PATH_INFO: `/` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:55:23.835353 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1458/2902] 10.222.0.25 () {54 vars in 921 bytes} [Fri Oct 28 09:55:23 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Oct 28 09:55:23.843259 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb410b2-f22d-4be9-ac02-3581062dc078 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:55:23.844157 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb410b2-f22d-4be9-ac02-3581062dc078 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:55:23.844157 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb410b2-f22d-4be9-ac02-3581062dc078 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:55:23.894266 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-6cb410b2-f22d-4be9-ac02-3581062dc078 None None] MFA Rules not processed for user `1db6b98e87e64584932447c82fefc3b7`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:55:23.895577 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-6cb410b2-f22d-4be9-ac02-3581062dc078 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:55:23.903123 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1445/2903] 10.222.0.25 () {58 vars in 1018 bytes} [Fri Oct 28 09:55:23 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 61 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:55:23.918823 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f83f92-7314-4f7b-b248-070bc327cbd1 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:55:23.941358 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f83f92-7314-4f7b-b248-070bc327cbd1 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:55:23.945958 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25fb4a77-21c8-41d0-b55b-9a954c763786 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:55:23.946529 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fb4a77-21c8-41d0-b55b-9a954c763786 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:55:23.946811 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fb4a77-21c8-41d0-b55b-9a954c763786 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:55:23.947122 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fb4a77-21c8-41d0-b55b-9a954c763786 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:55:23.956336 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25fb4a77-21c8-41d0-b55b-9a954c763786 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:55:23.957968 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25fb4a77-21c8-41d0-b55b-9a954c763786 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:55:23.968415 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1459/2904] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:55:23 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:55:24.084759 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2997d1-578c-4f90-8054-5f80e140c6d9 None None] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:55:24.085111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2997d1-578c-4f90-8054-5f80e140c6d9 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:55:24.085431 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2997d1-578c-4f90-8054-5f80e140c6d9 None None] PATH_INFO: `/` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:55:24.087081 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1446/2905] 10.222.0.25 () {54 vars in 921 bytes} [Fri Oct 28 09:55:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Oct 28 09:55:24.093333 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803f3fb1-e53d-4dbb-8e66-b6351dbaf520 None None] REQUEST_METHOD: `POST` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:55:24.093639 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803f3fb1-e53d-4dbb-8e66-b6351dbaf520 None None] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:55:24.093882 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803f3fb1-e53d-4dbb-8e66-b6351dbaf520 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:55:24.142145 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.auth.core [None req-803f3fb1-e53d-4dbb-8e66-b6351dbaf520 None None] MFA Rules not processed for user `688ceb201d304c2da5373c3cf36c2d79`. Rule list: `[]` (Enabled: `True`). {{(pid=81577) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:55:24.143559 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-803f3fb1-e53d-4dbb-8e66-b6351dbaf520 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:55:24.153305 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1460/2906] 10.222.0.25 () {58 vars in 1018 bytes} [Fri Oct 28 09:55:24 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 61 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Oct 28 09:55:24.164960 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c2997d1-578c-4f90-8054-5f80e140c6d9 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:55:24.177971 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c2997d1-578c-4f90-8054-5f80e140c6d9 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:55:24.181674 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ad75b25-99cb-4783-b9f1-d69a991cfb82 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:55:24.182961 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad75b25-99cb-4783-b9f1-d69a991cfb82 service neutron] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:55:24.183248 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad75b25-99cb-4783-b9f1-d69a991cfb82 service neutron] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:55:24.183523 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad75b25-99cb-4783-b9f1-d69a991cfb82 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:55:24.197250 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ad75b25-99cb-4783-b9f1-d69a991cfb82 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:55:24.199230 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ad75b25-99cb-4783-b9f1-d69a991cfb82 service neutron] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:55:24.212040 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1447/2907] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:55:24 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:55:34.138317 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=15 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:55:34.206053 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=15 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:55:44.201726 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-803f3fb1-e53d-4dbb-8e66-b6351dbaf520 None None] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:55:44.275122 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-803f3fb1-e53d-4dbb-8e66-b6351dbaf520 None None] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:55:44.280461 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e378a49d-6a66-44da-8465-79bfe8d50132 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:55:44.281112 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e378a49d-6a66-44da-8465-79bfe8d50132 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:55:44.281409 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e378a49d-6a66-44da-8465-79bfe8d50132 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:55:44.281666 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e378a49d-6a66-44da-8465-79bfe8d50132 admin admin] PATH_INFO: `/v3/users/55c55ff490f04152922385e619f79de9` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:55:44.282881 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e378a49d-6a66-44da-8465-79bfe8d50132 admin admin] RBAC: Authorizing `identity:delete_user(user_id=55c55ff490f04152922385e619f79de9)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:55:44.284826 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e378a49d-6a66-44da-8465-79bfe8d50132 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:55:44.343026 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e378a49d-6a66-44da-8465-79bfe8d50132 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '55c55ff490f04152922385e619f79de9'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:55:44.352140 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e378a49d-6a66-44da-8465-79bfe8d50132 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '55c55ff490f04152922385e619f79de9'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:55:44.353159 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e378a49d-6a66-44da-8465-79bfe8d50132 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '55c55ff490f04152922385e619f79de9'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:55:44.353891 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e378a49d-6a66-44da-8465-79bfe8d50132 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '55c55ff490f04152922385e619f79de9'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:55:44.363219 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e378a49d-6a66-44da-8465-79bfe8d50132 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '55c55ff490f04152922385e619f79de9'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:55:44.384787 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1461/2908] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:55:44 2022] DELETE /identity/v3/users/55c55ff490f04152922385e619f79de9 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:55:44.392303 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a1998a-b52d-494c-8fb0-2fda859ee9a6 None None] REQUEST_METHOD: `POST` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:55:44.392857 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a1998a-b52d-494c-8fb0-2fda859ee9a6 None None] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:55:44.393175 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98a1998a-b52d-494c-8fb0-2fda859ee9a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:55:44.422481 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.auth.core [None req-98a1998a-b52d-494c-8fb0-2fda859ee9a6 None None] MFA Rules not processed for user `8afbf6a033494d4cbbdea7a924ab5c63`. Rule list: `[]` (Enabled: `True`). {{(pid=81576) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Oct 28 09:55:44.424030 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-98a1998a-b52d-494c-8fb0-2fda859ee9a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:55:44.497752 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1448/2909] 10.222.0.25 () {58 vars in 958 bytes} [Fri Oct 28 09:55:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Oct 28 09:55:44.510167 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e378a49d-6a66-44da-8465-79bfe8d50132 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:55:44.514060 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e378a49d-6a66-44da-8465-79bfe8d50132 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:55:44.591742 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e378a49d-6a66-44da-8465-79bfe8d50132 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:55:44.596016 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd91265f-0900-440c-9890-1e70fe64c756 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:55:44.596658 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd91265f-0900-440c-9890-1e70fe64c756 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:55:44.596938 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd91265f-0900-440c-9890-1e70fe64c756 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:55:44.597212 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd91265f-0900-440c-9890-1e70fe64c756 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:55:44.608255 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd91265f-0900-440c-9890-1e70fe64c756 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:55:44.609707 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd91265f-0900-440c-9890-1e70fe64c756 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:55:44.625892 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1462/2910] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:55:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:55:45.021999 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a1998a-b52d-494c-8fb0-2fda859ee9a6 None None] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:55:45.024474 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-98a1998a-b52d-494c-8fb0-2fda859ee9a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:55:45.047349 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98a1998a-b52d-494c-8fb0-2fda859ee9a6 None None] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:55:45.055739 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d782984e-ea3c-4077-af7f-ba5dafd0e049 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:55:45.056856 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d782984e-ea3c-4077-af7f-ba5dafd0e049 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:55:45.057391 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d782984e-ea3c-4077-af7f-ba5dafd0e049 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:55:45.057766 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d782984e-ea3c-4077-af7f-ba5dafd0e049 admin admin] PATH_INFO: `/v3/projects/2a1784fc77a543e6adbc6194ceb272d2` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:55:45.060211 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d782984e-ea3c-4077-af7f-ba5dafd0e049 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2a1784fc77a543e6adbc6194ceb272d2)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:55:45.063771 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d782984e-ea3c-4077-af7f-ba5dafd0e049 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:55:45.132514 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-d782984e-ea3c-4077-af7f-ba5dafd0e049 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2a1784fc77a543e6adbc6194ceb272d2', 'request_id': 'req-d782984e-ea3c-4077-af7f-ba5dafd0e049'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:55:45.143357 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-d782984e-ea3c-4077-af7f-ba5dafd0e049 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2a1784fc77a543e6adbc6194ceb272d2', 'request_id': 'req-d782984e-ea3c-4077-af7f-ba5dafd0e049'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:55:45.153319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-d782984e-ea3c-4077-af7f-ba5dafd0e049 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2a1784fc77a543e6adbc6194ceb272d2', 'request_id': 'req-d782984e-ea3c-4077-af7f-ba5dafd0e049'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:55:45.169069 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-d782984e-ea3c-4077-af7f-ba5dafd0e049 admin admin] The token cache is being invalidate because project 2a1784fc77a543e6adbc6194ceb272d2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:55:45.169412 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-d782984e-ea3c-4077-af7f-ba5dafd0e049 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:55:45.170136 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-d782984e-ea3c-4077-af7f-ba5dafd0e049 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:55:45.172194 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1449/2911] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:55:45 2022] DELETE /identity/v3/projects/2a1784fc77a543e6adbc6194ceb272d2 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:55:54.616309 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=79,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:55:55.164982 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=54,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:56:12.345797 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd91265f-0900-440c-9890-1e70fe64c756 service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:56:12.347456 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-dd91265f-0900-440c-9890-1e70fe64c756 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:56:12.359754 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd91265f-0900-440c-9890-1e70fe64c756 service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:56:12.419990 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2874f48a-c936-4a86-a48c-e22c058a882e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:56:12.420618 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2874f48a-c936-4a86-a48c-e22c058a882e service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:56:12.420901 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2874f48a-c936-4a86-a48c-e22c058a882e service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:56:12.421177 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2874f48a-c936-4a86-a48c-e22c058a882e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:56:12.423493 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-2874f48a-c936-4a86-a48c-e22c058a882e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:56:12.434710 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2874f48a-c936-4a86-a48c-e22c058a882e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:56:12.436250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2874f48a-c936-4a86-a48c-e22c058a882e service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:56:12.446207 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1463/2912] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:56:12 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:56:22.441412 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=26 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:56:54.913283 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d782984e-ea3c-4077-af7f-ba5dafd0e049 admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:56:54.916111 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-d782984e-ea3c-4077-af7f-ba5dafd0e049 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:56:55.005027 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d782984e-ea3c-4077-af7f-ba5dafd0e049 admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:56:55.011426 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbbc6093-3656-4fa2-a80a-3742524b7aae service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '688ceb201d304c2da5373c3cf36c2d79', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:56:55.012186 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbbc6093-3656-4fa2-a80a-3742524b7aae service nova] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:56:55.012521 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbbc6093-3656-4fa2-a80a-3742524b7aae service nova] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:56:55.012975 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbbc6093-3656-4fa2-a80a-3742524b7aae service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:56:55.015180 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-cbbc6093-3656-4fa2-a80a-3742524b7aae service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:56:55.030319 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbbc6093-3656-4fa2-a80a-3742524b7aae service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:56:55.032670 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbbc6093-3656-4fa2-a80a-3742524b7aae service nova] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:56:55.051434 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1450/2913] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:56:54 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:57:00.293683 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2874f48a-c936-4a86-a48c-e22c058a882e service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:57:00.295223 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-2874f48a-c936-4a86-a48c-e22c058a882e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:57:00.307555 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2874f48a-c936-4a86-a48c-e22c058a882e service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:57:00.311983 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a87bcb-7879-49d9-8759-e46424e7bc16 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '1db6b98e87e64584932447c82fefc3b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:57:00.312609 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a87bcb-7879-49d9-8759-e46424e7bc16 service neutron] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:57:00.312894 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a87bcb-7879-49d9-8759-e46424e7bc16 service neutron] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:57:00.313169 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a87bcb-7879-49d9-8759-e46424e7bc16 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:57:00.314998 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-48a87bcb-7879-49d9-8759-e46424e7bc16 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:57:00.384255 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48a87bcb-7879-49d9-8759-e46424e7bc16 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:57:00.385914 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48a87bcb-7879-49d9-8759-e46424e7bc16 service neutron] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:57:00.397153 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1464/2914] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:57:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:57:04.165839 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbbc6093-3656-4fa2-a80a-3742524b7aae service nova] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:57:04.167916 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-cbbc6093-3656-4fa2-a80a-3742524b7aae service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:57:04.274786 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbbc6093-3656-4fa2-a80a-3742524b7aae service nova] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:57:04.284412 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50eb845-4975-4a64-802f-3456bd23a2d3 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3f1d95049d2d4593bf52670b1a0035fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:57:04.285783 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50eb845-4975-4a64-802f-3456bd23a2d3 service placement] REQUEST_METHOD: `GET` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:57:04.286318 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50eb845-4975-4a64-802f-3456bd23a2d3 service placement] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:57:04.286884 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50eb845-4975-4a64-802f-3456bd23a2d3 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:57:04.290171 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-e50eb845-4975-4a64-802f-3456bd23a2d3 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:57:04.311763 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50eb845-4975-4a64-802f-3456bd23a2d3 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:57:04.313450 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50eb845-4975-4a64-802f-3456bd23a2d3 service placement] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:57:04.328640 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1451/2915] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:57:04 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 163 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:57:08.070296 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a87bcb-7879-49d9-8759-e46424e7bc16 service neutron] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:57:08.072555 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-48a87bcb-7879-49d9-8759-e46424e7bc16 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:57:08.096410 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a87bcb-7879-49d9-8759-e46424e7bc16 service neutron] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:57:08.106913 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11d2df3-0908-429f-a288-9b4b347a05d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:57:08.107857 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11d2df3-0908-429f-a288-9b4b347a05d6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:57:08.108741 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11d2df3-0908-429f-a288-9b4b347a05d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:57:08.109278 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11d2df3-0908-429f-a288-9b4b347a05d6 admin admin] PATH_INFO: `/v3/users/f14c3f34010b41ef9be2b9fd2aa98cd9` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:57:08.110785 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e11d2df3-0908-429f-a288-9b4b347a05d6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f14c3f34010b41ef9be2b9fd2aa98cd9)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:57:08.114076 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e11d2df3-0908-429f-a288-9b4b347a05d6 admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:57:08.195371 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e11d2df3-0908-429f-a288-9b4b347a05d6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f14c3f34010b41ef9be2b9fd2aa98cd9'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:57:08.206528 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e11d2df3-0908-429f-a288-9b4b347a05d6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f14c3f34010b41ef9be2b9fd2aa98cd9'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:57:08.207658 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e11d2df3-0908-429f-a288-9b4b347a05d6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f14c3f34010b41ef9be2b9fd2aa98cd9'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:57:08.208297 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e11d2df3-0908-429f-a288-9b4b347a05d6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f14c3f34010b41ef9be2b9fd2aa98cd9'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:57:08.226472 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-e11d2df3-0908-429f-a288-9b4b347a05d6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f14c3f34010b41ef9be2b9fd2aa98cd9'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:57:08.257267 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1465/2916] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:57:08 2022] DELETE /identity/v3/users/f14c3f34010b41ef9be2b9fd2aa98cd9 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:57:08.478588 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50eb845-4975-4a64-802f-3456bd23a2d3 service placement] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:57:08.498235 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50eb845-4975-4a64-802f-3456bd23a2d3 service placement] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:57:08.571174 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e96ae6-1a7b-4850-ba1f-1810d5992edc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:57:08.572134 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e96ae6-1a7b-4850-ba1f-1810d5992edc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:57:08.572611 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e96ae6-1a7b-4850-ba1f-1810d5992edc admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:57:08.573099 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e96ae6-1a7b-4850-ba1f-1810d5992edc admin admin] PATH_INFO: `/v3/projects/937954dbeac043c29259ada2924c4953` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:57:08.575185 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e96ae6-1a7b-4850-ba1f-1810d5992edc admin admin] RBAC: Authorizing `identity:delete_project(project_id=937954dbeac043c29259ada2924c4953)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:57:08.578747 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e96ae6-1a7b-4850-ba1f-1810d5992edc admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:57:08.610944 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=100 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:57:08.653653 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-99e96ae6-1a7b-4850-ba1f-1810d5992edc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '937954dbeac043c29259ada2924c4953', 'request_id': 'req-99e96ae6-1a7b-4850-ba1f-1810d5992edc'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:57:08.660660 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-99e96ae6-1a7b-4850-ba1f-1810d5992edc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '937954dbeac043c29259ada2924c4953', 'request_id': 'req-99e96ae6-1a7b-4850-ba1f-1810d5992edc'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:57:08.670834 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-99e96ae6-1a7b-4850-ba1f-1810d5992edc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '937954dbeac043c29259ada2924c4953', 'request_id': 'req-99e96ae6-1a7b-4850-ba1f-1810d5992edc'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:57:08.683961 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-99e96ae6-1a7b-4850-ba1f-1810d5992edc admin admin] The token cache is being invalidate because project 937954dbeac043c29259ada2924c4953 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:57:08.684303 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-99e96ae6-1a7b-4850-ba1f-1810d5992edc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:57:08.685078 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-99e96ae6-1a7b-4850-ba1f-1810d5992edc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:57:08.686844 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1452/2917] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:57:08 2022] DELETE /identity/v3/projects/937954dbeac043c29259ada2924c4953 => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Oct 28 09:57:08.691341 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11d2df3-0908-429f-a288-9b4b347a05d6 admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:57:08.692839 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-e11d2df3-0908-429f-a288-9b4b347a05d6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:57:08.707904 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11d2df3-0908-429f-a288-9b4b347a05d6 admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:57:08.775777 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01358834-22b4-4bd2-8c4d-d5c3f49a5c1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:57:08.776587 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01358834-22b4-4bd2-8c4d-d5c3f49a5c1c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:57:08.776871 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01358834-22b4-4bd2-8c4d-d5c3f49a5c1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:57:08.777151 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01358834-22b4-4bd2-8c4d-d5c3f49a5c1c admin admin] PATH_INFO: `/v3/users/4de7d277bca0435b9eaafc9503fc9119` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:57:08.778379 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01358834-22b4-4bd2-8c4d-d5c3f49a5c1c admin admin] RBAC: Authorizing `identity:delete_user(user_id=4de7d277bca0435b9eaafc9503fc9119)` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:57:08.780360 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01358834-22b4-4bd2-8c4d-d5c3f49a5c1c admin admin] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:57:08.787212 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:57:08.841229 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-01358834-22b4-4bd2-8c4d-d5c3f49a5c1c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4de7d277bca0435b9eaafc9503fc9119'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:57:08.850075 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-01358834-22b4-4bd2-8c4d-d5c3f49a5c1c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4de7d277bca0435b9eaafc9503fc9119'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:57:08.850732 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-01358834-22b4-4bd2-8c4d-d5c3f49a5c1c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4de7d277bca0435b9eaafc9503fc9119'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:57:08.851905 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-01358834-22b4-4bd2-8c4d-d5c3f49a5c1c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4de7d277bca0435b9eaafc9503fc9119'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:57:08.860139 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.notifications [None req-01358834-22b4-4bd2-8c4d-d5c3f49a5c1c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4de7d277bca0435b9eaafc9503fc9119'} {{(pid=81577) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:57:08.886087 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1466/2918] 10.222.0.25 () {58 vars in 1224 bytes} [Fri Oct 28 09:57:08 2022] DELETE /identity/v3/users/4de7d277bca0435b9eaafc9503fc9119 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:57:09.090051 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e96ae6-1a7b-4850-ba1f-1810d5992edc admin admin] Authenticating user token {{(pid=81576) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:57:09.091769 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.fernet_utils [None req-99e96ae6-1a7b-4850-ba1f-1810d5992edc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81576) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:57:09.107677 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e96ae6-1a7b-4850-ba1f-1810d5992edc admin admin] Validating token access rules against request {{(pid=81576) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:57:09.201261 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a6378a-006d-473b-a322-4289d773d654 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8afbf6a033494d4cbbdea7a924ab5c63', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ac54d153685c4b2288a7ed215d0e24d2', 'project_domain_id': 'default', 'roles': ['member', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81576) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:57:09.202209 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a6378a-006d-473b-a322-4289d773d654 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:57:09.202721 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a6378a-006d-473b-a322-4289d773d654 admin admin] SCRIPT_NAME: `/identity` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:57:09.203188 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a6378a-006d-473b-a322-4289d773d654 admin admin] PATH_INFO: `/v3/projects/09213a17ab164ade9dcf58d2399f1790` {{(pid=81576) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:57:09.204676 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75a6378a-006d-473b-a322-4289d773d654 admin admin] RBAC: Authorizing `identity:delete_project(project_id=09213a17ab164ade9dcf58d2399f1790)` {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:57:09.206990 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75a6378a-006d-473b-a322-4289d773d654 admin admin] RBAC: Authorization granted {{(pid=81576) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:57:09.275131 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-75a6378a-006d-473b-a322-4289d773d654 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '09213a17ab164ade9dcf58d2399f1790', 'request_id': 'req-75a6378a-006d-473b-a322-4289d773d654'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:57:09.285671 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-75a6378a-006d-473b-a322-4289d773d654 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '09213a17ab164ade9dcf58d2399f1790', 'request_id': 'req-75a6378a-006d-473b-a322-4289d773d654'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:57:09.298493 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-75a6378a-006d-473b-a322-4289d773d654 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '09213a17ab164ade9dcf58d2399f1790', 'request_id': 'req-75a6378a-006d-473b-a322-4289d773d654'} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:57:09.314824 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-75a6378a-006d-473b-a322-4289d773d654 admin admin] The token cache is being invalidate because project 09213a17ab164ade9dcf58d2399f1790 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81576) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Oct 28 09:57:09.315282 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-75a6378a-006d-473b-a322-4289d773d654 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:57:09.316188 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG keystone.notifications [None req-75a6378a-006d-473b-a322-4289d773d654 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81576) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Oct 28 09:57:09.317887 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: [pid: 81576|app: 0|req: 1453/2919] 10.222.0.25 () {58 vars in 1233 bytes} [Fri Oct 28 09:57:09 2022] DELETE /identity/v3/projects/09213a17ab164ade9dcf58d2399f1790 => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Oct 28 09:57:18.875197 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=23,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:57:19.311984 dv-cinder-zed-3-iscsi devstack@keystone.service[81576]: DEBUG dbcounter [-] [81576] Writing DB stats keystone:SELECT=68,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81576) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Oct 28 09:59:04.663833 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01358834-22b4-4bd2-8c4d-d5c3f49a5c1c admin admin] Authenticating user token {{(pid=81577) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Oct 28 09:59:04.665417 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-01358834-22b4-4bd2-8c4d-d5c3f49a5c1c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:59:04.752097 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01358834-22b4-4bd2-8c4d-d5c3f49a5c1c admin admin] Validating token access rules against request {{(pid=81577) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Oct 28 09:59:04.828699 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c426bb-74f1-44e8-9bf7-2d31df5df815 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3f1d95049d2d4593bf52670b1a0035fa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '455d1c826b974bc38497d18a516867c7', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81577) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Oct 28 09:59:04.829345 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c426bb-74f1-44e8-9bf7-2d31df5df815 service placement] REQUEST_METHOD: `GET` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Oct 28 09:59:04.829629 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c426bb-74f1-44e8-9bf7-2d31df5df815 service placement] SCRIPT_NAME: `/identity` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Oct 28 09:59:04.829904 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c426bb-74f1-44e8-9bf7-2d31df5df815 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81577) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Oct 28 09:59:04.831833 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.fernet_utils [None req-f5c426bb-74f1-44e8-9bf7-2d31df5df815 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81577) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Oct 28 09:59:04.845250 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5c426bb-74f1-44e8-9bf7-2d31df5df815 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Oct 28 09:59:04.846926 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5c426bb-74f1-44e8-9bf7-2d31df5df815 service placement] RBAC: Authorization granted {{(pid=81577) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Oct 28 09:59:04.859370 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: [pid: 81577|app: 0|req: 1467/2920] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Oct 28 09:59:04 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 196 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Oct 28 09:59:14.854359 dv-cinder-zed-3-iscsi devstack@keystone.service[81577]: DEBUG dbcounter [-] [81577] Writing DB stats keystone:SELECT=58 {{(pid=81577) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}